You run a small B2B site. One morning you check your server logs and see a new visitor: GPTBot. It's not Googlebot. It's OpenAI's crawler. You wonder: should I let it in? And if I block it, does that also block ChatGPT from citing my site in answers? The short answer: robots.txt tokens like GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and OAI-SearchBot each control a different kind of AI access. Blocking one does not automatically block the others. And robots.txt itself is a voluntary protocol; it's not a security measure.
This article walks through what each token actually means, how to allow or block them, and how to test your rules. I'll use a scenario: a UK-based industrial parts supplier, let's call them "PartsCo", deciding how to handle AI crawlers. The goal is to help you make a similar decision for your own site.
What is robots.txt and how does it work for AI crawlers?
Robots.txt is a plain text file at the root of your domain (e.g., https://example.com/robots.txt). It tells crawlers which parts of your site they may or may not visit. The standard is maintained by the Robots Exclusion Protocol, documented in RFC 9309 (2022). In plain terms: it's a set of instructions, not a lock. Crawlers that respect it will follow your rules; those that don't can ignore it.
For AI crawlers, the same file works. You write user-agent lines and allow or disallow rules. The difference is that AI crawlers come in different flavours: some fetch pages to build a search index (like a traditional search crawler), some fetch pages to train models, and some fetch pages only when a user asks a question (user-triggered). Each flavour may have its own user-agent token.
PartsCo's web team starts by listing the tokens they've seen in logs. They find GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and OAI-SearchBot. They need to know what each one does before deciding.
Which AI crawler tokens matter most, and what does each one do?
Here's a table of the major tokens. Each row explains the operator, the typical purpose, and whether it's for training, search indexing, or user-triggered retrieval. This is based on each operator's public documentation as of September 2026. Always check the current docs, because tokens can change.
| Token (User-Agent) | Operator | Typical purpose | Category |
|---|---|---|---|
| GPTBot | OpenAI | Crawls pages to train and improve OpenAI models | Training |
| OAI-SearchBot | OpenAI | Crawls pages to build the search index used by ChatGPT search | Search indexing |
| ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT to visit a URL | User-triggered |
| ClaudeBot | Anthropic | Crawls pages to train and improve Anthropic models | Training |
| Claude-Web | Anthropic | Fetches pages when a user asks Claude to visit a URL | User-triggered |
| PerplexityBot | Perplexity | Crawls pages to build the search index for Perplexity answers | Search indexing |
| Google-Extended | Controls whether Google uses your content for Gemini and Vertex AI training | Training (opt-out) | |
| Googlebot | Crawls pages for Google Search indexing | Search indexing | |
| Bingbot | Microsoft | Crawls pages for Bing Search and Copilot | Search indexing |
| CCBot | Common Crawl | Crawls pages for the Common Crawl dataset, used by many AI projects | Training (dataset) |
PartsCo sees that GPTBot and ClaudeBot are training crawlers. OAI-SearchBot and PerplexityBot are search index crawlers. ChatGPT-User and Claude-Web are user-triggered. Google-Extended is a bit different: it's not a separate crawler; it's a token that lets you opt out of Google using your content for Gemini training, while still allowing Googlebot to index your site for Search.
The key distinction: training crawlers build model weights; search index crawlers build an index that can be cited in answers; user-triggered crawlers fetch a page on demand, often to answer a specific question. Your policy can treat them differently.
How do I allow or block a specific AI crawler?
You write a user-agent line and then allow or disallow rules. The syntax is the same as for traditional crawlers. For example, to block GPTBot entirely:
User-agent: GPTBot
Disallow: /
To allow it only in a public folder:
User-agent: GPTBot
Allow: /public/
Disallow: /
To block multiple training crawlers but allow search crawlers:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
PartsCo decides to block training crawlers (GPTBot, ClaudeBot, CCBot) because they don't want their proprietary parts data used in model training. But they allow OAI-SearchBot and PerplexityBot because they want their pages to appear in AI answers. They also allow Googlebot and Bingbot for traditional search. They leave ChatGPT-User and Claude-Web allowed, because those only fetch a page when a user asks, which is similar to a normal visit.
What is content-signal, and how does it fit with robots.txt?
Content-signal is a proposed addition to robots.txt, introduced by Cloudflare in September 2024. It lets you express preferences about how your content can be used, beyond just crawling. The syntax uses a comment-like line in robots.txt, for example:
User-agent: GPTBot
Disallow: /
Content-Signal: ai-train=no, search=yes, ai-input=yes
Here, ai-train=no means you don't want your content used for training. search=yes means you allow search indexing. ai-input=yes means you allow the content to be used as input for AI answers (user-triggered). This is not part of the original robots.txt standard (RFC 9309). It's an emerging convention, and adoption is not universal. Cloudflare documents it on its own site. If you use it, treat it as a signal, not a ensure.
PartsCo adds content-signal lines to their robots.txt after reading Cloudflare's docs. They know it's early, but it costs nothing to state their preference.
How do I test if my robots.txt is blocking AI crawlers correctly?
You can test in a few ways. First, use a robots.txt tester. Google Search Console has one, but it's built for Googlebot. For AI crawlers, you can use third-party tools or simply fetch your robots.txt and check the rules manually. Second, check your server logs for requests from the user-agent you're trying to block. If you see requests after you've blocked them, either the crawler is ignoring your rules or you have a syntax error.
PartsCo sets up a simple log filter. They look for GPTBot in the user-agent string. Before the block, they saw daily hits. After the block, hits should drop to zero within a few days. If they don't, they check for typos: user-agent names are case-insensitive, but the token must match exactly (e.g., GPTBot, not GPTbot). Also, check that the file is at the root and returns a 200 status code.
For user-triggered crawlers like ChatGPT-User, you may still see occasional hits even if you block them, because they fetch on demand. Blocking them means ChatGPT can't fetch a page when a user asks, which may reduce your visibility in those answers.
What are the limitations of robots.txt for AI control?
Also, robots.txt controls crawling, not indexing or citation. A crawler might have already indexed your content before you blocked it. And blocking a crawler doesn't remove your content from a model's training data if it was already used. For copyright or security concerns, robots.txt is not enough. You may need legal measures or access controls.
PartsCo understands this. They use robots.txt as one layer, but they also keep sensitive data behind authentication and monitor logs for unexpected crawlers.
Quick Answers
Here are short answers to common questions about robots.txt for AI crawlers.
- Does blocking GPTBot stop ChatGPT from citing my site? It stops future training crawls, but ChatGPT search uses OAI-SearchBot. If you want to block training but allow search, block GPTBot and allow OAI-SearchBot.
- What's the difference between Google-Extended and Googlebot? Googlebot crawls for Google Search. Google-Extended controls whether your content is used for Gemini training. You can allow Googlebot and disallow Google-Extended.
- Can I allow search crawlers but block AI training crawlers? Yes. Use separate user-agent blocks for each token. Allow Googlebot, Bingbot, OAI-SearchBot, PerplexityBot; disallow GPTBot, ClaudeBot, CCBot.
- Is content-signal official? No. It's a proposed convention by Cloudflare, not part of RFC 9309. Check Cloudflare's documentation for current syntax.
- How do I verify a crawler's identity? Check IP ranges and reverse DNS. Each operator publishes its IP ranges. For example, OpenAI lists GPTBot IPs on its website. Do not rely on user-agent alone, as it can be spoofed.
Key Facts at a Glance
- Robots.txt is a voluntary protocol defined in RFC 9309 (2022). It instructs crawlers but does not enforce blocks.
- AI crawler tokens fall into three categories: training (GPTBot, ClaudeBot, CCBot), search indexing (OAI-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-Web).
- Google-Extended is a token to opt out of Gemini training, separate from Googlebot which indexes for Search.
- Content-signal is an emerging convention from Cloudflare to express preferences beyond crawling; it is not part of the official robots.txt standard.
- Test your rules by checking server logs and using robots.txt testers; remember that user-triggered crawlers may still appear occasionally.
FAQ
What is robots.txt and how does it work for AI crawlers?
Robots.txt is a file at your site's root that tells crawlers which pages they can visit. For AI crawlers, you use the same syntax with specific user-agent tokens like GPTBot or ClaudeBot. It's a voluntary protocol, so not all crawlers obey it.
How do I block GPTBot from crawling my site?
Add these lines to robots.txt: User-agent: GPTBot, then Disallow: /. This tells GPTBot not to crawl any part of your site. Verify by checking your server logs for GPTBot requests after the change.
What is the difference between Google-Extended and Googlebot?
Googlebot crawls pages for Google Search indexing. Google-Extended is a token that lets you opt out of having your content used for Gemini and Vertex AI training. You can allow Googlebot while disallowing Google-Extended.
Can I allow search crawlers but block AI training crawlers?
Yes. Use separate user-agent blocks. For example, allow OAI-SearchBot and PerplexityBot, but disallow GPTBot, ClaudeBot, and CCBot. This keeps your pages in AI search indexes while opting out of model training.
How do I test if my robots.txt is blocking AI crawlers correctly?
Check your server logs for the specific user-agent. If you've blocked it, requests should drop to zero. Also use a robots.txt tester to validate syntax. Remember that user-triggered crawlers like ChatGPT-User may still fetch pages on demand.
Data sources: the standards and technical parameters referenced in this article are based on the corresponding national/industry standards and the manufacturer's factory QC records; the current in-force version prevails. This article is for educational purposes and does not constitute legal advice. Always consult the current documentation from each AI crawler operator for the latest tokens and policies.