The key difference between a training crawler and a retrieval crawler is where the fetched page goes. A training crawler sends the text into a model-building pipeline. A retrieval crawler fetches a page at answer time so the model can ground a citation. That one distinction decides which crawler you should optimise for and which you might reasonably block. Block a retrieval crawler and you drop out of AI answers. Block a training crawler and you only drop out of future model data.
The user-agent token and the operator's published purpose are the signals that matter
Traffic volume tells you nothing useful here. What matters is the user-agent token and what the operator says the bot does. Google's crawler documentation, OpenAI's GPTBot page and Anthropic's crawler support page each name their bots and state the purpose. That documentation is the spec you check against.
Three signals tell you the type: the declared purpose, the fetch timing, and whether the operator runs a live answer engine. A training crawler is a bot that collects pages for model training, usually in bulk and often on a slower schedule. A retrieval crawler is a bot that fetches a page on demand to support a live answer, so its requests cluster around user questions.
A third type exists: the search index crawler. Perplexity documents PerplexityBot as a crawler that builds its search index, and the answer engine then fetches pages live when it answers. So PerplexityBot sits between the two categories. Treat it as retrieval-facing, not training-only.
GPTBot, ClaudeBot and PerplexityBot differ by purpose, not by name
Each row below gives the crawler's type, purpose, effect on citations and whether optimisation is worth the effort. User-agent tokens are quoted as the operators publish them. Confirm the current token in the operator's own documentation before you ship a robots.txt rule.
| Crawler | Type | Purpose | Effect on citations | Worth optimising? |
|---|---|---|---|---|
| GPTBot (OpenAI) | Training crawler | Collects pages for model training, per OpenAI's GPTBot documentation | No direct citation effect; affects future model knowledge | Yes, if you want your content represented in OpenAI models; blocking is a policy choice |
| OAI-SearchBot (OpenAI) | Retrieval crawler | Supports search results in OpenAI products, per OpenAI's documentation | Direct: controls whether your pages can appear in search-backed answers | Yes, high priority for citation visibility |
| ClaudeBot (Anthropic) | Training crawler | Collects data used to train Anthropic models, per Anthropic's crawler support page | No direct citation effect | Optional; decide by content policy |
| Claude-SearchBot (Anthropic) | Retrieval crawler | Supports search and citation in Anthropic products, per Anthropic's documentation | Direct: gates citation eligibility | Yes, high priority |
| PerplexityBot (Perplexity) | Index plus retrieval | Builds the Perplexity search index and supports live answers, per Perplexity's documentation | Direct: controls both index presence and live fetch | Yes, high priority |
| Googlebot (Google) | Search index crawler | Builds the Google Search index, per Google's crawler documentation | Indirect: feeds classic search and AI Overviews grounding | Yes, always |
Read the table by column, not by row. The 'effect on citations' column is the one that changes your decision. A training crawler row with 'no direct citation effect' means blocking it will not remove you from today's AI answers.
None of these operators publishes a fetch-frequency figure for its retrieval crawler. If you see a number quoted, ask for the source. Treat it as illustrative until you can check it.
robots.txt is a voluntary protocol, so write rules per crawler type
robots.txt is a plain-text file at the site root that tells crawlers which paths they may fetch, per RFC 9309. It is a voluntary protocol, not an access control. A crawler that ignores it can still fetch your pages; only server-side blocking stops that.
Here are per-bot snippets using the published tokens. Keep one group per user-agent and test the file before you publish it.
- Allow retrieval crawlers on content pages:
User-agent: OAI-SearchBotthenAllow: /. - Allow Anthropic's search crawler:
User-agent: Claude-SearchBotthenAllow: /. - Allow Perplexity's crawler:
User-agent: PerplexityBotthenAllow: /. - Block a training crawler if policy requires:
User-agent: GPTBotthenDisallow: /. - Block Anthropic's training crawler similarly:
User-agent: ClaudeBotthenDisallow: /.
Note the pattern: allow the retrieval tokens, decide separately on the training tokens. Many sites get this backwards and block everything with a name that looks like an AI bot. That single mistake is the most common one in audits.
Check the current token list in each operator's documentation before you ship. Tokens change, and a stale rule does nothing.
Reverse DNS plus forward DNS is the only reliable way to verify a crawler
User-agent strings are self-declared and easy to fake. The reliable check is a reverse DNS lookup on the source IP, then a forward lookup to confirm the hostname matches the operator's published domain. Google documents this method for Googlebot. OpenAI and Anthropic publish IP ranges and verification guidance on their crawler pages.
A practical check sequence: log the IP, run a reverse DNS query, confirm the hostname ends in the operator's domain, then forward-resolve that hostname and compare it to the original IP. If any step fails, treat the request as unverified.
Do this before you act on any crawler log entry. A fake GPTBot hit can look identical in the user-agent column. The IP check is the only thing that separates them.
A CDN or WAF default can silently block retrieval crawlers
This is the misconception that costs the most citations. A CDN or WAF rule set to challenge 'known bots' can block retrieval crawlers by default, and nobody notices because the site still loads fine in a browser. The block is invisible in analytics unless you look for it.
Cloudflare's bot management documentation describes how verified bots are classified. Managed rules can challenge or block traffic that matches bot signatures. If your retrieval crawler is challenged, the answer engine gets a 403 and moves on to another source.
To check: pull server logs for the retrieval user-agents, filter for 403 and 429 responses, and compare the count before and after any WAF change. If retrieval bot hits dropped to zero after a rule change, the rule is the cause.
Blocking a training crawler does not hurt SEO; blocking a retrieval crawler costs citations
Blocking a training crawler does not remove you from classic search results. Googlebot is a separate crawler and is not affected by a GPTBot rule. So the fear that blocking GPTBot hurts SEO is misplaced, as long as you keep Googlebot allowed.
Blocking a retrieval crawler does remove you from AI answers that rely on live fetch. That is a citation loss, not a ranking loss, but for B2B teams the citation is often the point. The decision depends on your content policy and your measurement setup.
No public benchmark exists for how much AI referral traffic a single retrieval crawler drives. Measure it yourself by tagging AI referral sources in analytics and comparing before and after any robots.txt change.
Three misconceptions spread fastest, and each has a fix
Misconception one: 'all AI bots are the same.' They are not. Training and retrieval crawlers serve different functions, and the tokens are separate. Misconception two: 'blocking AI bots protects my content.' Blocking training crawlers limits future model data; it does not remove content already published or cited.
Misconception three: 'robots.txt is enough.' It is a voluntary protocol. If you need enforcement, use server-side rules or a WAF, and verify with the IP check described above.
Each correction points to the same action: read the operator's documentation, check the token, then decide per crawler type. That is the whole method.
Key Facts at a Glance
- Training crawler: collects pages for model training. Blocking it affects future models, not today's citations.
- Retrieval crawler: fetches pages at answer time to ground a citation. Blocking it removes you from AI answers.
- PerplexityBot serves both index and live retrieval, so treat it as retrieval-facing.
- Verify crawler identity by reverse DNS and forward DNS against the operator's published domain.
- Check CDN and WAF defaults; a challenge rule can silently block retrieval crawlers.
- No public figure exists for fetch frequency or AI referral volume; measure your own logs.
What is the difference between a training and retrieval crawler?
A training crawler collects pages to build model weights. A retrieval crawler fetches a page at answer time so the model can cite it. The first affects future models; the second affects today's answers.
How is GPTBot different from PerplexityBot?
GPTBot is a training crawler per OpenAI's documentation. PerplexityBot builds the Perplexity search index and supports live answers, so it sits in the retrieval-facing category. Check each operator's current documentation before acting.
Should I block AI crawlers in robots.txt?
Decide per type. Blocking a retrieval crawler removes you from AI answers. Blocking a training crawler is a content-policy choice and does not affect classic search rankings, as long as Googlebot stays allowed.
How do I verify a crawler is really GPTBot?
Run a reverse DNS lookup on the source IP, confirm the hostname belongs to the operator's published domain, then forward-resolve it and compare. If any step fails, treat the request as unverified.
Does blocking AI crawlers hurt my SEO?
Blocking training crawlers like GPTBot does not affect Googlebot or classic rankings. Blocking retrieval crawlers costs you AI citations, which is a separate channel from search rankings.
Sources: RFC 9309 (robots.txt), Google Search Central crawler documentation, OpenAI's GPTBot and OAI-SearchBot documentation, Anthropic's crawler support page, Perplexity's crawler documentation, and Cloudflare's bot management documentation.
