Most AI crawler advice makes one mistake: it treats every bot as the same kind of thing. A training crawler collects text to build a language model. A retrieval crawler fetches pages while a user waits for an answer, so the model can quote or link them. That gap decides whether a visit can ever turn into a citation.
Most guides list the user-agents and stop. The useful question is which bots can put your page in front of a real user, and which only feed a model you'll never see. Those are two different jobs, and they need two different robots.txt decisions.
This piece starts with the spec — the control token, the user-agent, the fetch — then compares each major crawler in one table. It ends with a checklist you can run before you change anything.
Robots.txt is a request, not a lock, so read the spec first
Start with the fetch, not the name. A user-agent is the string a bot sends in its HTTP request to say who it is. Robots.txt matches on that string.
Robots.txt is defined in RFC 9309, published by the IETF in September 2022. A crawler is a program that fetches pages automatically and follows links. A retrieval crawler fetches a page because a user just asked a question, and the answer may cite that page.
Two limits matter before any table. RFC 9309 is voluntary — a bot can ignore robots.txt, and some do. And a user-agent string is self-declared, so anyone can send GPTBot in a request header.
Put simply: robots.txt is a signal to well-behaved crawlers. It is not a lock.
A training crawler feeds a model; a retrieval crawler feeds an answer
A training crawler gathers pages into a corpus used to train or fine-tune a model. The fetch happens once, offline from any user question, and the page's text may end up in model weights.
A retrieval crawler sits inside the answer path. When a user asks something, the system fetches or re-fetches pages, reads them, and may quote a passage with a link. The fetch is tied to a query, so the page can appear as a citation.
That is the whole distinction, and it drives everything else. Training crawls feed a model you cannot see. Retrieval crawls can put a URL in an answer a user reads.
One caveat: the line is not always clean. Some operators use the same crawler for both jobs, or run a separate retrieval fetcher that reuses the training user-agent. Perplexity is the clearest case, covered below.
The purpose column, not the name column, tells you what a bot is for
The table below is the core of this article. Each row gives the type, the stated purpose, the effect on citations, and whether optimizing for that bot is worth the effort.
| Crawler (user-agent) | Type | Stated purpose | Effect on citations | Worth optimizing? |
|---|---|---|---|---|
| GPTBot | Training | OpenAI's crawler for training and improving models; OpenAI documents it as blockable in robots.txt | None directly — no citation, no click | Low, unless you want training data excluded |
| OAI-SearchBot | Retrieval | OpenAI's crawler for surfacing sites in ChatGPT search results; separate from GPTBot | Can lead to a citation and a click | High — this is the one to allow |
| ChatGPT-User | Retrieval (on demand) | Fetches a page when a user asks ChatGPT to visit a URL or when browsing is triggered | Can support an answer with a live fetch | Medium — keep pages fetchable |
| ClaudeBot | Training | Anthropic's crawler for model training data | None directly | Low for citation goals |
| Claude-User | Retrieval (on demand) | Fetches pages on behalf of a Claude user request | Can feed a cited answer | Medium |
| PerplexityBot | Mixed — retrieval and training | Perplexity documents PerplexityBot as used for search results and for improving its models | Can drive citations; also feeds training | High if you want citations — see the ambiguity note |
| Perplexity-User | Retrieval (on demand) | Fetches a page when a Perplexity user asks about it | Can support a live answer | Medium |
| Googlebot | Search indexing | Google Search's crawler; feeds the search index | Indirect — AI features in Google draw on search | High — keep it allowed |
| Google-Extended | Control token, not a crawler | A robots.txt token that opts a site out of Gemini training and grounding; it does not fetch pages itself | None by itself | Decision only — see below |
| CCBot | Training | Common Crawl's crawler; builds an open corpus many models train on | None directly | Low for citation goals |
| Applebot-Extended | Control token, not a crawler | Opts out of Apple's AI training use; Applebot itself does search indexing | None by itself | Decision only |
Read the table by column three, not column one. The purpose tells you what the fetch is for. If the purpose is "training data," a citation is not on the table.
User-agent names above follow each operator's own documentation. Confirm the exact string on the operator's page before you edit robots.txt, because names can change.
Blocking GPTBot does not remove you from ChatGPT search
This pair trips people up because the two bots are not the same kind of thing. GPTBot is OpenAI's training crawler. Blocking it does not remove your site from ChatGPT search, because that path uses OAI-SearchBot and ChatGPT-User.
PerplexityBot is the harder case. Perplexity documents it as used both for search results and for improving its models. So blocking it can cut a citation path and a training feed at the same time.
If citations are your goal, allow PerplexityBot. If you have a policy against training use, Perplexity's documentation does not offer a clean split here — check Perplexity's own crawler docs for the current position before you decide.
Do not assume a robots.txt rule aimed at GPTBot covers PerplexityBot. Each user-agent needs its own line, and RFC 9309 matching is literal.
Google-Extended and Applebot-Extended are tokens, not crawlers
Google-Extended is a control token, not a crawler. It appears in robots.txt as a user-agent, but it does not send its own fetch requests. Google documents it as a way to control whether your content is used for Gemini training and grounding.
That means a log file will never show a Google-Extended visit. If you go looking for it in access logs, you will find Googlebot and nothing else.
The same pattern applies to Applebot-Extended. It is a token that sets a training preference; Applebot does the fetching.
So the decision is real, but the verification is different. You confirm a token by reading your robots.txt, not by grepping a log.
Name each bot explicitly, because a wildcard can catch the ones you meant to allow
Start from the goal. If you want citations, allow the retrieval crawlers and keep pages reachable. If you want to exclude training, block the training bots by name.
A minimal allow for retrieval looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
Allow: /
A block for training looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
Two rules to remember. First, a User-agent: * group applies to any bot not named elsewhere, so a broad disallow can catch retrieval bots you meant to allow. Name each bot explicitly.
Second, robots.txt does not enforce itself. A crawler that ignores it will still fetch. There is no public enforcement mechanism for the token, and spoofing a user-agent is trivial.
Check the log and the IP range, because the user-agent string can be faked
Robots.txt is a request; the log is the evidence. Open your server access log and filter on the user-agent string. That tells you which bots fetched, how often, and which URLs.
Then check the source. A self-declared user-agent can be faked, so confirm the request comes from the operator's published IP ranges. OpenAI, for example, publishes its crawler IP ranges in its documentation.
If the user-agent says GPTBot but the IP does not match OpenAI's range, treat it as unverified. Do not block on the string alone.
To verify a token like Google-Extended, you cannot use the log. Read your robots.txt and check the token is present and spelled correctly.
Expect a gap between a robots.txt change and a change in log traffic. If a site is new, expect crawl activity to grow over the first weeks before you draw conclusions.
The Short Version
- Training crawlers collect text for model building. Retrieval crawlers fetch pages at answer time and can produce a citation.
- GPTBot is training; OAI-SearchBot and ChatGPT-User are the retrieval paths for ChatGPT. Blocking GPTBot alone does not remove you from ChatGPT search.
- PerplexityBot is documented as both a search and a training crawler, so a block cuts both uses.
- Google-Extended and Applebot-Extended are control tokens, not crawlers. They never appear in access logs.
- Robots.txt is voluntary under RFC 9309. Verify real visits in the log and by IP range.
Key Facts at a Glance
- RFC 9309 (IETF, September 2022) defines robots.txt matching.
- GPTBot, ClaudeBot and CCBot are training crawlers; OAI-SearchBot, ChatGPT-User, Claude-User and Perplexity-User are retrieval paths.
- PerplexityBot is documented as used for both search and model improvement.
- Google-Extended and Applebot-Extended are opt-out tokens, not fetchers.
- A user-agent string is self-declared; confirm with the operator's published IP ranges.
- Robots.txt signals intent but does not enforce it.
What is the difference between a training and retrieval crawler?
A training crawler collects pages to build or improve a model, offline from any user question. A retrieval crawler fetches pages at answer time so the model can quote or link them. Only the retrieval path can produce a citation.
Should I block GPTBot in robots.txt?
Block it only if you want to exclude your content from OpenAI's training data. Blocking GPTBot does not remove your site from ChatGPT search, because that path uses OAI-SearchBot and ChatGPT-User.
Is PerplexityBot a training or retrieval crawler?
Perplexity documents PerplexityBot as used for search results and for improving its models, so it sits in both categories. Blocking it can cut a citation path and a training feed at once.
Does blocking AI crawlers hurt my SEO?
It can, if you block a retrieval crawler. Google Search indexing runs on Googlebot, so keep that allowed. Blocking training bots like GPTBot or CCBot does not change your Google rankings directly.
How do I check which AI crawlers visit my site?
Filter your server access log on the user-agent string, then confirm the request comes from the operator's published IP ranges. For control tokens like Google-Extended, read your robots.txt instead — they never appear in logs.
Primary documents: OpenAI: Overview of OpenAI crawlers and Google Search Central: Google's common crawlers.