A web lead at a mid-sized B2B supplier opens a support ticket: "ChatGPT recommended our competitor for a product we invented. Did we block it by accident?" The answer often sits in one file. Robots.txt is the plain-text file at the root of a site that tells crawlers which paths they may fetch. Each AI vendor runs its own crawler, and each crawler has its own token. Put simply: the token you allow or block decides whether that vendor's system can read your pages at all.
The short answer to the ticket: check which token the vendor uses for search versus training. GPTBot is OpenAI's training crawler, while OAI-SearchBot is the one tied to ChatGPT search results. Blocking GPTBot stops training. Blocking OAI-SearchBot can remove you from ChatGPT answers. Most accidental disappearances come from blocking the second one.
One robots.txt file now controls two very different things: training and live answers
Classic search worked one way: crawl, index, rank, click. AI answer engines add a step. They retrieve pages at question time, then summarise and cite. A blocked page cannot be retrieved, and a page that is not retrieved cannot be cited. That is the whole chain: crawl, index, retrieve, cite.
Robots.txt sits at the front of that chain. It is voluntary but widely respected. The Robots Exclusion Protocol was formalised as RFC 9309 by the IETF in September 2022. That document says a crawler should fetch the file before crawling and obey the rules it finds. It also says a group that matches a crawler's token overrides the wildcard group.
The practical catch is granularity. One wildcard rule treats every crawler the same. Named tokens let you split the policy: allow retrieval bots, block training bots. That split is the core skill this page covers.
Each vendor publishes its own token, and the vendor page is the authority
A user-agent token is the name a crawler sends in its HTTP request, and the name you match in robots.txt. Vendors publish their tokens in their own documentation, so treat the vendor page as the authority, not a third-party list.
| Token | Owner | What it does | Blocking it usually means |
|---|---|---|---|
| GPTBot | OpenAI | Fetches pages for model training | Your pages are excluded from future training data |
| OAI-SearchBot | OpenAI | Supports ChatGPT search results | Your pages can drop out of ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetches a page when a user asks for it | Live page fetches inside a chat can fail |
| ClaudeBot | Anthropic | Training and general crawling | Pages excluded from Anthropic training crawls |
| PerplexityBot | Perplexity | Indexing for Perplexity answers | Pages may not appear in Perplexity answers |
| Google-Extended | Controls use of content in Gemini training and grounding | Content not used for those products; Search ranking is unaffected | |
| Googlebot | Google Search crawling and indexing | Pages drop out of Google Search entirely |
Two rows deserve a warning. Google-Extended is not a crawler you can block at fetch time in the usual sense; Google documents it as a control token for Gemini and Vertex AI use. Blocking it does not remove a page from Google Search. Googlebot does. Mixing those two up is the most common expensive mistake.
ClaudeBot's exact split between training and retrieval is worth checking against Anthropic's current documentation before you write a policy.
Content-Signal adds a stated preference, not a lock
Content-Signal is an addition to robots.txt proposed by Cloudflare in September 2024. It is a comment line that states preferences in three categories: search, ai-input and ai-train. Because it starts with a hash, standard parsers ignore it. Only systems that read the convention act on it.
A typical line looks like this: Content-Signal: search=yes, ai-input=yes, ai-train=no. Read it as a stated preference, not a lock. It gives you a way to say "index me, quote me in answers, don't train on me" in one line that sits next to your user-agent rules.
There is no public figure for how many crawlers honour it. Treat it as a signal that reinforces your user-agent rules, never as a replacement for them.
To allow answer engines but block training, write separate user-agent groups
Start from intent. If your goal is citations in AI answers, you want retrieval bots in and training bots out. If your goal is privacy, you may want both out and accept the citation loss. Write the policy down before you touch the file, because the file is hard to debug later.
Here is a copy-ready pattern. Adjust the paths to your site; the token names are the part that matters.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: * Allow: /
One rule from RFC 9309 matters here: groups are matched by token, and an explicit group wins over the wildcard. So the wildcard allow at the bottom does not undo the GPTBot block. Keep groups separate and never merge tokens into one line.
If you also use content-signal, add the line once, near the top. Keep it consistent with the user-agent rules. A file that says ai-train=no and also allows GPTBot sends two different messages.
Test the policy in three layers: syntax, fetch behaviour and business outcome
Skip none of the three layers. Each one catches a different failure.
- Syntax: paste the file into Google Search Console's robots.txt report, or fetch
/robots.txtdirectly and read the status code. A 200 with plain text is what you want. - Fetch behaviour: watch server logs for the tokens you care about. A crawler that respects the file will stop requesting blocked paths.
- Business outcome: run the same prompt against the AI engine before and after the change. Note whether your domain still appears as a cited source.
A quick self-check list before you publish: every token spelled exactly as the vendor publishes it; no token merged into a comma list; wildcard group present; content-signal line consistent with the groups; sitemap line intact. That is five checks and takes about ten minutes.
One more habit: date the change. If citations drop three weeks later, you want to know what you edited and when. This is the same discipline as any other change to a site's crawl surface.
Blocking the wrong token costs answers today, not training data tomorrow
Blocking a training bot costs you future training data. Blocking a retrieval bot costs you answers today. Those are different losses, and the second one is usually what a business notices first.
There is no public benchmark that quantifies citation loss per blocked token. The mechanism is easy to reason about, though: if the retrieval step cannot fetch the page, the model has nothing to quote, and a competitor's page fills the slot.
For a B2B site, that slot is often a buying question: "who supplies X in region Y". Losing it is a lead-flow problem, not an SEO vanity metric. That is why you should check the retrieval tokens first when a client reports missing AI citations.
The ticket that opened this page usually ends the same way: the site blocked OAI-SearchBot months ago during a privacy review, then forgot. The fix is one group edit and a re-test. The lesson is to name the token you are changing and write down why.
Key Facts at a Glance
- Robots.txt follows RFC 9309, published by the IETF in September 2022.
- GPTBot and OAI-SearchBot are different OpenAI tokens with different effects.
- Google-Extended controls Gemini use, not Google Search ranking.
- Content-Signal is a Cloudflare-proposed comment convention from September 2024.
- Explicit user-agent groups override the wildcard group.
- Verify with Search Console, server logs and a before/after prompt test.
Should you block GPTBot or allow it?
Block it only if you do not want your pages in future OpenAI training data. It has no direct effect on ChatGPT search visibility, which is governed by OAI-SearchBot.
What is the content-signal directive?
A comment line in robots.txt, proposed by Cloudflare in September 2024, that states preferences for search, ai-input and ai-train. Standard parsers ignore it; only systems that read the convention act on it.
Does blocking Google-Extended affect AI Overviews?
Google documents Google-Extended as a control for Gemini training and grounding. Google Search crawling and ranking remain governed by Googlebot. Check Google's current documentation before relying on this.
How do you allow AI search bots but block training bots?
Write separate user-agent groups: allow OAI-SearchBot and PerplexityBot, disallow GPTBot, ClaudeBot and Google-Extended. Keep a wildcard group and make sure the explicit groups stay separate.
How do you test your robots.txt for AI crawlers?
Check syntax in Search Console's robots.txt report, watch server logs for the tokens you care about, and run the same prompt against the AI engine before and after the change to see whether your domain is still cited.
If you want to compare policies, list your own dimensions first: which engines matter for your buyers, which content is sensitive, and how fast you need to see the effect. Then test each token against those dimensions. That beats copying a competitor's file, because their business is not yours.