An in-house SEO lead at a mid-size B2B distributor opens Search Console on a Tuesday, sees GPTBot in the crawl stats, and asks a simple question: does the robots.txt file already tell that bot to stay out? The short answer is that robots.txt is a plain-text file at the site root that tells named crawlers which paths they may fetch, and each AI-related token in it is a separate user-agent string with its own rules. Nothing in the file forces compliance. The tokens only work when a crawler chooses to read and honour them.
What does robots.txt actually control when an AI crawler arrives?
A crawler first fetches /robots.txt from the host, then checks its own user-agent token against the groups in the file. The protocol is defined in RFC 9309, published by the IETF in September 2022, and that document is the reference point for parsing rules. A user-agent token is the name a crawler sends in its HTTP request header; the robots.txt group matching that name decides what happens next.
The file is fetched before any page on the site is requested. If the fetch returns a 404, per RFC 9309 the crawler may treat the site as unrestricted. If the fetch returns a 5xx, the crawler should treat the site as fully disallowed, at least for a period, because the rules cannot be read. Those two failure modes are not the same, and confusing them is common.
Put simply, robots.txt is a request, not a lock. It does not authenticate, it does not encrypt, and it does not stop a bot that ignores it. A site that needs enforcement has to look at rate limiting, WAF rules, or authentication, which sit outside the protocol.
Which token belongs to which AI crawler?
Each vendor publishes its own token and its own purpose. The table below lists the main ones in active use as of 2026 and what each is documented to do. Treat the purpose column as what the vendor says the bot does, not as a promise about what happens to the fetched text afterwards.
| Token | Owner | Documented purpose | What a Disallow affects |
|---|---|---|---|
| GPTBot | OpenAI | Training data collection for OpenAI models | Blocks that collection path; does not affect ChatGPT search results |
| OAI-SearchBot | OpenAI | Fetching pages for ChatGPT search answers | Removes the site from that search surface |
| ChatGPT-User | OpenAI | User-initiated browsing when a person asks ChatGPT to open a page | Blocks that on-demand fetch |
| ClaudeBot | Anthropic | Training data collection for Claude models | Blocks that collection path |
| Claude-User | Anthropic | User-initiated fetch during a Claude conversation | Blocks that on-demand fetch |
| PerplexityBot | Perplexity | Indexing pages used to answer Perplexity queries | Removes the site from Perplexity answers |
| Google-Extended | Controls use of content in Gemini training and grounding | Does not affect Google Search crawl or ranking | |
| Googlebot | Google Search crawling and indexing | Removes the site from Google Search | |
| CCBot | Common Crawl | Building the public Common Crawl dataset | Excludes the site from that dataset |
The distinction that matters most is training versus retrieval. GPTBot and ClaudeBot are documented as training collectors. OAI-SearchBot and PerplexityBot exist to fetch pages that will be used to answer a live query. Blocking the first pair protects content from training use; blocking the second pair removes the site from the answer surface itself.
Google-Extended is the odd one. Google documents it as a control for Gemini training and grounding, separate from Googlebot. Disallowing Google-Extended does not change how Google Search crawls or ranks the site. That separation is why some sites block Google-Extended but keep Googlebot fully allowed.
How do Allow, Disallow and wildcards resolve?
The syntax is small. User-agent names the crawler, Allow and Disallow give paths, and * matches any sequence of characters. A line starting with # is a comment. A blank line ends a group. RFC 9309 sets out the matching rules, including the convention that the most specific matching rule wins.
Specificity is measured by path length. If a file contains Disallow: / and Allow: /blog/, a request for /blog/post-1 matches the longer Allow rule and is permitted. A request for /pricing matches only the Disallow and is blocked. This is why a single blanket Disallow followed by a few Allow lines can carve out a useful public section while keeping the rest closed.
Wildcards add reach. Disallow: /*?sessionid= blocks any path containing that query string. Disallow: /*.pdf$ blocks PDFs, where $ anchors the match to the end of the URL. Crawlers that follow RFC 9309 should support * and $; older or non-conforming parsers may not.
One trap: a group with only a User-agent line and no rules is treated as allowing everything. Another: a User-agent: * group applies to every crawler that is not named elsewhere, including AI bots you have never heard of. If the goal is to block unnamed bots, the wildcard group is where that happens.
What does content-signal add on top of robots.txt?
Content-Signal is a set of directives placed in robots.txt, proposed through the IETF draft process, that expresses preferences about how content may be used rather than whether it may be fetched. The three directives are search=yes|no, ai-input=yes|no and ai-train=yes|no. They sit in the same file as the crawl rules but describe a different intent.
An example line looks like this: Content-Signal: search=yes, ai-input=no, ai-train=no. Read plainly, it says the site welcomes search indexing, does not want its text fed into live AI answers, and does not want it used for training. Whether a given crawler honours that line depends on the crawler. There is no public enforcement mechanism, and the draft status means the wording can still change.
Practically, Content-Signal is a signal of intent, not a control. It helps when a vendor chooses to respect it and does nothing when a vendor does not. Treat it as a documented preference that sits alongside Disallow rules, not as a replacement for them. Sites that want a hard block still need Disallow lines or network-level controls.
Where should the file live, and how is it fetched?
The file must be served at the root of the host, at https://example.com/robots.txt, with a 200 status and a text/plain content type. RFC 9309 requires the path to be exactly /robots.txt; a file at /blog/robots.txt is not read for the whole site. Crawlers may follow redirects, but a redirect chain adds latency and a loop can cause the fetch to fail.
Subdomains and separate hosts each need their own file. shop.example.com and www.example.com are different hosts, and a rule in one does not apply to the other. Sites behind a CDN should confirm the CDN is not serving a cached copy that differs from origin.
If the fetch fails with a 5xx, RFC 9309 says crawlers should assume the site is disallowed until a later fetch succeeds. That behaviour is worth knowing before a maintenance window: a brief origin outage can look, from the crawler's side, like a deliberate block.
How do you block or allow specific AI crawlers without breaking search?
The safe pattern is to name each bot in its own group and leave Googlebot and Bingbot untouched. A copy-paste starting point for a site that wants search indexing but no AI training or AI answer use looks like this:
User-agent: GPTBotthenDisallow: /User-agent: ClaudeBotthenDisallow: /User-agent: CCBotthenDisallow: /User-agent: OAI-SearchBotthenDisallow: /User-agent: PerplexityBotthenDisallow: /User-agent: Google-ExtendedthenDisallow: /User-agent: *thenAllow: /
That snippet is a policy choice, not a default. Blocking OAI-SearchBot and PerplexityBot removes the site from those answer surfaces, which is usually the opposite of what a B2B team wants. A more common split is to allow search-oriented bots and block training-only bots. The exact list depends on which surfaces the business cares about.
There is no public figure for how much AI-referred traffic a typical B2B site receives, so I would not promise a traffic outcome either way. The decision should rest on the site's own analytics and on the vendor documentation for each token.
How do you test a change before it goes live?
Test the file in staging first. Fetch it with a plain HTTP client and read the raw response, not the rendered page. Check the status code, the content type, and that no HTML wrapper has crept in. A robots.txt file that begins with <!DOCTYPE html> is a soft-404 page, and crawlers will treat it as no rules at all.
Then check the parser. Google publishes a robots.txt tester in Search Console, and Bing documents its own tester. Both read the file the way their crawler does. For AI bots, the practical check is to watch the server log for the token after the change and confirm the request rate for that token drops, or that the bot stops fetching disallowed paths.
If the site is new, expect crawl activity to grow over the first weeks regardless of robots.txt changes, so compare a period before the edit with a period after it rather than a single day. Keep the diff small: one group at a time makes it easier to see which change had which effect.
Key Facts at a Glance
- Robots.txt is fetched from
/robots.txton each host, per RFC 9309 (IETF, September 2022). - GPTBot and ClaudeBot are training collectors; OAI-SearchBot and PerplexityBot serve live AI answers.
- Google-Extended controls Gemini training and grounding, not Google Search.
- Content-Signal adds
search,ai-inputandai-trainpreferences; it is a signal, not a lock. - A 5xx on the robots.txt fetch is read as a full disallow until the next successful fetch.
- Most specific rule wins; longer path beats shorter path.
How do I block GPTBot in robots.txt?
Add a group with User-agent: GPTBot and Disallow: /. That tells OpenAI's training collector to stay out of the whole site. It does not affect OAI-SearchBot, which is a separate token for ChatGPT search.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is documented by OpenAI as a training data collector. OAI-SearchBot fetches pages to answer ChatGPT search queries. Blocking the first limits training use; blocking the second removes the site from that answer surface.
Does robots.txt stop AI from scraping my site?
No. Robots.txt is a voluntary protocol. A crawler that ignores it will still fetch pages. Enforcement needs rate limiting, a WAF, or authentication at the network level.
What is content-signal and should I use it?
Content-Signal is a robots.txt directive set that expresses preferences for search, AI input and AI training use. It is an IETF draft. Use it to state intent; do not rely on it as a hard block.
How do I allow Google-Extended but block training?
Leave Google-Extended out of any Disallow group, or set it to Allow: /. Then add separate Disallow groups for GPTBot, ClaudeBot and CCBot. Googlebot stays unaffected either way.