A training crawler collects pages to build or fine-tune a model. A retrieval crawler fetches a page the moment someone asks a question, so the model can quote it in the answer. That one difference decides whether a fetch in your logs can turn into a citation today, or never.
Plenty of site owners mix the two up. They block one crawler, assume they fixed the other problem, and then wonder why their pages never show up in AI answers. This piece fixes the common mistakes and shows how to check the reality in your own logs.
The short version
- Training crawlers build models. Retrieval crawlers answer live questions. Same page, different job.
- Blocks are per-token, not per-company. Blocking GPTBot does not block ChatGPT-User.
- Retrieval hits are the ones that can become citations in the same session.
- Check three things together: user-agent, published IP range, reverse DNS. A user-agent string alone proves nothing.
- Robots.txt is a request, not a wall. Some crawlers ignore it.
A crawler's job, not its name, tells you what it is
A crawler is a program that requests web pages on its own, with no human clicking. What it was sent to do separates the two types.
Training crawlers work in bulk and on a schedule. They pull large sets of pages, often from sitemaps, and store the text for model training. The fetch you see today may be used months later, or never, and you cannot trace it back to a specific user.
Retrieval crawlers work per query. Someone asks a question, the system searches, fetches the most useful pages, and reads them to build the answer. The fetch happens seconds before the answer appears. That is why retrieval traffic matters for citations: it is the pipeline that quotes you.
Put simply, training is a library build. Retrieval is a librarian running to the shelf while you wait.
OpenAI, Google, Perplexity and Anthropic each run more than one token
OpenAI publishes separate tokens for separate jobs. GPTBot is the training and content-indexing crawler. ChatGPT-User is the token used when a user asks ChatGPT to open a page. OAI-SearchBot is the token used for search-result fetching. Blocking GPTBot does not touch the other two.
Google runs a similar split. Googlebot is the search crawler and feeds Google Search. Google-Extended is a robots.txt token that controls whether your content can be used for Gemini model training and grounding, and it does not affect Search inclusion.
Perplexity documents PerplexityBot for its search index and Perplexity-User for user-triggered fetches. Same pattern: one token for indexing, one for live retrieval.
Anthropic documents ClaudeBot for training and Claude-User for user-triggered fetches. Common Crawl runs CCBot, which builds an open dataset that many models train on.
On the standards side, sitemaps.org defines the sitemap protocol, and RFC 9309 defines the Robots Exclusion Protocol. Read both before you write policy.
Compare the major AI crawlers in one table
Use this as a working sheet. Purpose and citation effect are the two columns that should drive your decision.
| Crawler | Type and purpose | Effect on citation | Worth optimizing? |
|---|---|---|---|
| GPTBot (OpenAI) | Training and content indexing | Indirect; no live query link | Only if you want model-training inclusion |
| ChatGPT-User (OpenAI) | Retrieval, user-triggered | Direct; the fetched page can be quoted in that answer | Yes, if you want citations in ChatGPT |
| OAI-SearchBot (OpenAI) | Retrieval for search results | Direct; supports answer grounding | Yes |
| Googlebot | Search indexing | Feeds Google Search; indirect for AI answers | Yes, for search visibility |
| Google-Extended | Training and grounding control token | Does not affect Search inclusion | Decide by policy, not by ranking |
| PerplexityBot | Search index building | Indirect; feeds the index | Yes, if you want Perplexity visibility |
| Perplexity-User | Retrieval, user-triggered | Direct; can be cited in the answer | Yes |
| ClaudeBot (Anthropic) | Training and indexing | Indirect | Policy call |
| Claude-User (Anthropic) | Retrieval, user-triggered | Direct; the fetched page can be quoted | Yes |
| CCBot (Common Crawl) | Open dataset for training | Indirect; downstream models depend on it | Only if you want open-corpus inclusion |
The practical rule: if the token contains "User" or "Search", it is retrieval. If it is a plain bot name, it is usually training or indexing.
Check three signals before you trust a crawler hit
User-agent strings are easy to fake, so never trust them alone. Match three things together.
First, the user-agent string. Second, the source IP against the operator's published IP list, like the JSON files OpenAI and Google publish for their crawlers. Third, reverse DNS, which should resolve back to the operator's domain.
If the IP is not on the published list, treat the hit as unverified. It may be a scraper pretending to be GPTBot. Anthropic and Perplexity publish their own crawler documentation if you need a starting point.
To do this at scale, filter your access log by user-agent, group by /24 subnet, and export the count per week. That shows you which tokens are actually fetching, and how often.
There is no single public benchmark that covers crawl volume for every operator, so compare your own weekly counts instead of chasing a number someone else published.
Write robots.txt per token, and test before you push
Robots.txt lives at the root of your domain and lists which user-agents may fetch which paths. RFC 9309 is the current spec. The sitemaps.org protocol covers the sitemap file itself.
Two mistakes are common. The first is blocking a retrieval token by accident while testing a training block. The second is blocking Googlebot while believing you only blocked Google-Extended. Both remove your pages from the pipelines you wanted.
Write the file in blocks, one user-agent per block, and comment each line with the reason. Then test with a crawler that respects robots.txt in a staging environment before you push it live.
Remember that robots.txt is a voluntary standard. Some scrapers ignore it entirely. It shapes the behaviour of the major compliant operators, not every bot on the internet.
Decide your goal first, then build the checklist
Decide first what you actually want. If the goal is citations in AI answers, the retrieval tokens matter most, because they fetch your page during the answer. If the goal is model-training inclusion, the training tokens matter, and the payoff is slow and hard to measure.
Then implement in this order: write your robots.txt policy per token, verify hits with IP and reverse DNS, review the log weekly for the first month, and adjust. Keep a written record of which tokens you allow and why, so the decision survives staff changes.
Last, do not treat this as a one-time setup. Operators add and rename tokens. A short quarterly review of the official crawler documentation from OpenAI, Google, Anthropic, and Perplexity keeps your policy current.
Key Facts at a Glance
- Training crawlers build models; retrieval crawlers fetch pages during a live query.
- GPTBot is training and indexing. ChatGPT-User and OAI-SearchBot are retrieval.
- PerplexityBot builds the search index. Perplexity-User fetches for a live question.
- Verify crawler identity by user-agent, published IP range, and reverse DNS together.
- Robots.txt follows RFC 9309 and is voluntary; some scrapers ignore it.
- Retrieval tokens are the ones that can turn a fetch into a citation in the same session.
Questions to ask a GEO tool vendor before you buy
If you are evaluating a GEO or AI-visibility tool, ask the vendor these questions before you sign. The answers tell you whether the tool can actually see retrieval traffic or only guesses at it.
- Does it separate training tokens from retrieval tokens in the report?
- Does it verify crawler IPs against the operator's published list, or trust the user-agent string?
- Does it show per-page fetch frequency, so you can see which pages retrieval crawlers actually pull?
- Does it track the token names that exist today, and how often is that list updated?
- Does it export raw log lines, so you can audit the claims yourself?
What is the difference between a training crawler and a retrieval crawler?
A training crawler collects pages in bulk to build or fine-tune a model. A retrieval crawler fetches a page during a live query so the model can quote it in the answer. The first is a library build; the second is a librarian running to the shelf.
Does blocking GPTBot also block ChatGPT from citing my site?
No. GPTBot is one token. ChatGPT-User and OAI-SearchBot are separate tokens used for user-triggered and search fetches. You have to set policy for each one.
How can I tell if an AI crawler really visited my site?
Match three things: the user-agent string, the source IP against the operator's published IP list, and reverse DNS. If the IP is not on the published list, treat the hit as unverified.
Should I allow PerplexityBot to crawl my content?
That depends on your goal. If you want visibility in Perplexity answers, allowing both PerplexityBot and Perplexity-User keeps you in the index and available for live retrieval. Blocking them removes you from both.
What is llms.txt and how does it relate to AI crawlers?
llms.txt is a proposed convention for giving AI systems a short, structured summary of your site. It is not part of the Robots Exclusion Protocol (RFC 9309) and is not a standard. Treat it as an experiment, and keep robots.txt as your main control.