A site owner we'll call Maya runs a small industrial parts business. One morning she notices her server logs full of requests from something called GPTBot. She wants AI search to find her pages, but she doesn't want her product descriptions used to train a model for free. She opens robots.txt, sees a list of user-agent tokens she doesn't recognise, and closes the file again. This article walks through what each token means, what happens at each stage of the crawl-to-cite pipeline, and how to write rules you can actually verify.
The direct answer: robots.txt is a plain-text file at your domain root that tells crawlers which paths they may request. Each AI company uses its own user-agent token, and each token maps to a different job - training, search indexing, or live retrieval. Blocking a token only affects that token. To control AI use, you need to know which token does what, then write rules per token and check the result in your logs.
What does robots.txt actually control?
Robots.txt is a voluntary exclusion protocol, first published as an internet draft by Martijn Koster in 1994 and now standardised in RFC 9309 (IETF, September 2022). The file sits at https://example.com/robots.txt and lists user-agent groups with allow and disallow rules.
A user-agent token is the name a crawler sends in its HTTP request header. When a bot requests a page, your server sees a string like GPTBot/1.2. The token is the part before the slash. Robots.txt rules match on that token.
Two limits matter. First, robots.txt is voluntary - RFC 9309 says so directly. A crawler that ignores it faces no technical barrier. Second, robots.txt only stops requests. If a page was already crawled and cached elsewhere, a new rule doesn't remove that copy.
Which token does which job?
Each AI company publishes its own crawler documentation. The table below lists the main tokens and their stated purpose. Check the vendor's page for the current in-force version, because these tokens change.
| Token | Operator | Stated job | Robots.txt respected? |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI models | Yes, per OpenAI docs |
| OAI-SearchBot | OpenAI | Builds the search index behind ChatGPT search features | Yes, per OpenAI docs |
| ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT to open a URL | Yes, per OpenAI docs |
| ClaudeBot | Anthropic | Crawls content for model training | Yes, per Anthropic docs |
| Claude-User | Anthropic | Fetches a page on behalf of a user in real time | Yes, per Anthropic docs |
| PerplexityBot | Perplexity | Indexes pages to answer user queries with citations | Yes, per Perplexity docs |
| Google-Extended | Controls use of content in Gemini training and grounding. Does not affect Google Search ranking or inclusion. | Yes, per Google Search Central | |
| Googlebot | Standard search indexing | Yes |
Two patterns stand out. OpenAI and Anthropic each run a training crawler and a live-fetch crawler. Perplexity runs one crawler that does both indexing and citation. Google separates search from AI training entirely through Google-Extended.
The practical rule: blocking GPTBot does not block OAI-SearchBot. They're separate tokens with separate jobs. If you want AI search visibility but not training use, you block one and allow the other.
How does the crawl-to-cite pipeline work?
The pipeline has four stages, and robots.txt only touches the first.
Crawl. A bot sends an HTTP GET to a URL. Your server logs the request. Robots.txt decides whether the bot is allowed to make that request in the first place.
Index. The bot stores the page content in a database. This is where training data and search indexes diverge. Training crawlers store text for model weights. Search crawlers store text for retrieval.
Retrieve. When a user asks a question, the AI system searches its index for relevant passages. Perplexity, ChatGPT search and Bing chat all do this at query time.
Cite. The system generates an answer and links to source pages. Citation depends on the page being in the index and being judged relevant. If you blocked the crawler at stage one, the page never reaches stage four.
This is why blocking GPTBot and allowing OAI-SearchBot is a coherent choice: you remove training use but keep the retrieval path open.
What does the content-signal directive add?
Content-signal is a proposed extension to robots.txt that lets you state preferences beyond allow and disallow. It was drafted by the Content Signals Policy group and is documented at contentsignals.org. The syntax looks like this:
Content-Signal: ai-train=no, search=yes, ai-input=yes
The three signals mean: ai-train controls whether content may be used for model training. search controls whether it may appear in search results. ai-input controls whether it may be used as input to an AI system at query time.
Content-signal is not yet an IETF standard. It's a published proposal. Adoption depends on each crawler operator choosing to honour it. Treat it as a preference statement, not an enforcement mechanism. For hard blocks, use disallow rules.
How do you write the rules?
Robots.txt uses groups. A group starts with one or more User-agent lines, followed by Allow and Disallow lines. A blank line ends the group. The wildcard * matches any sequence of characters; $ anchors to the end of a URL.
Three common scenarios:
Block all AI training, allow AI search:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Allow everything:
User-agent: *
Allow: /
Block one directory from all AI crawlers:
User-agent: GPTBot
Disallow: /internal/
Keep in mind that a more specific path rule overrides a broader one. Allow: /blog/ beats Disallow: / for URLs under /blog/.
How do you verify it's working?
Robots.txt is easy to get wrong. Here's a check you can run:
- Open your robots.txt in a browser. If it returns a 404, your rules don't exist.
- Confirm the file sits at the domain root, not a subfolder.
- Check for typos in user-agent tokens.
GPTbotandGPTBotare not the same to a strict parser. - Use Google Search Console's robots.txt tester to validate syntax.
- After 48 hours, grep your server logs for the token you blocked. If it still appears, either the crawler ignores the rule or your syntax is wrong.
- Check the response code you send. A 403 or 429 tells the crawler something different from a 200 with a disallow rule.
Log analysis is the only real proof. A robots.txt rule is a request, not a promise. Some crawlers honour it within hours; others may take longer or ignore it entirely.
What mistakes are common?
Blocking Googlebot by accident. A rule like User-agent: * blocks every crawler, including search engines. If you only want to block AI training, name the tokens explicitly.
Disallow: /
Confusing Google-Extended with Googlebot. Google's own documentation states that Google-Extended does not affect Search ranking or inclusion. Blocking it only opts you out of Gemini training and grounding.
Assuming one rule covers all AI. Each vendor uses a different token. There is no single User-agent: AI line that works.
Forgetting that robots.txt is public. Anyone can read it, including competitors. It's not a security tool.
Leaving the file stale. Tokens change. OpenAI, Anthropic, Google and Perplexity all publish current lists. Review your file at least twice a year.
Key Facts at a Glance
- Robots.txt is a voluntary protocol (RFC 9309). It requests, it does not enforce.
- GPTBot, ClaudeBot and PerplexityBot are separate tokens with separate jobs.
- Google-Extended controls AI training only; it does not affect Search.
- Content-signal is a published proposal, not yet an IETF standard.
- Verify with server logs after 48 hours, not with the file alone.
Does blocking GPTBot hurt my Google rankings?
No. Google Search uses Googlebot, not GPTBot. Blocking GPTBot only removes your content from OpenAI's training crawl. Google's ranking systems do not look at whether GPTBot can access your site.
Can I allow AI search but block AI training?
Yes. Allow OAI-SearchBot, PerplexityBot and Googlebot, then disallow GPTBot and ClaudeBot. This keeps your pages in AI search indexes while opting out of training crawls. Check each vendor's current documentation because token behaviour can change.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls content for model training. OAI-SearchBot builds the index behind ChatGPT search features. They are different tokens from the same company, and robots.txt treats them separately.
Does content-signal replace robots.txt rules?
No. Content-signal is an additional line inside robots.txt that states preferences. It is not an IETF standard and depends on crawler operators honouring it. Use disallow rules for hard blocks.
How long before a robots.txt change takes effect?
It depends on the crawler. Some re-read robots.txt on every visit; others cache it for up to 24 hours. Check your server logs after 48 hours to confirm the change is live. There is no universal figure for this.
If you want to compare robots.txt configurations for your own site, list the dimensions that matter to you - which AI features you want visibility in, which training uses you want to opt out of, and how much of your content is behind login walls - and work through the table above line by line.