Training vs Retrieval Crawlers: What People Get Wrong

Here's the mistake I see most: treating every AI crawler as one thing, a bot that either helps you or hurts you. That's wrong. A training crawler collects pages to build or update a model's weights. A retrieval crawler fetches a page at answer time so the engine can quote it. Only the retrieval path can put your content in front of a user today.

Blocking GPTBot does not block OAI-SearchBot. Blocking PerplexityBot removes your page from Perplexity's answer pool. If you want citations, keep the retrieval path open.

The key spec is the user-agent token, not the brand name

What matters is the crawler's declared purpose, not the company that runs it. A user-agent is the string a bot sends in the HTTP request header, and robots.txt matches on that string. Google's robots.txt documentation says rules are matched by user-agent token, so one token can be allowed while another from the same company is blocked.

That's why "allow OpenAI" and "allow GPTBot" are different commands. OpenAI publishes separate tokens for separate jobs: GPTBot for model training, OAI-SearchBot for search results, and ChatGPT-User for user-triggered visits. Each one needs its own decision.

Check the vendor's official crawler page before writing any rule, because tokens do change. OpenAI's documentation lists GPTBot, OAI-SearchBot and ChatGPT-User as distinct agents with distinct IP ranges. Anthropic lists ClaudeBot for training and Claude-SearchBot for search. Perplexity lists PerplexityBot for search.

Training crawlers build models; retrieval crawlers answer questions

A training crawler collects pages into a dataset used to build or update a model. The page itself is not shown to a user at that moment. A retrieval crawler fetches a page when a user asks a question, so the engine can read it and cite it in the answer.

The difference shows up in timing. Training crawls run in batches, often over weeks, and the result is baked into a future model version. Retrieval crawls run near real time, triggered by a query, and the result appears in that answer.

Citations come from the retrieval path. If a retrieval bot can't fetch the page, the engine has nothing to quote, so your site can't be cited for that query. That's the core reason the two types deserve separate rules.

One row needs care. Google-Extended is not a crawler. Google's documentation describes it as a robots.txt token that lets site owners control whether their content is used for Gemini training and grounding. Googlebot still crawls the page regardless of what Google-Extended says.

Your server logs tell you which bot actually visits

Logs are the only honest answer to "which bot actually visits us". A server log records the user-agent string and the requesting IP for every hit. Matching the user-agent against the vendor's published list tells you which bot is which.

Don't trust the user-agent string alone, because it can be faked. Verify the IP against the vendor's published range. OpenAI publishes its crawler IP ranges, and Perplexity publishes its own list. A hit that claims to be PerplexityBot from an IP outside that range is not PerplexityBot.

To check, pull a week of logs and count hits per token. If OAI-SearchBot is absent but GPTBot is present, your retrieval path is not being exercised, and your citation chances in OpenAI answers are limited. If PerplexityBot never appears, your page isn't in Perplexity's pool.

There's no public figure for how often each bot crawls a typical site. It depends on site size, update frequency and crawl budget, so treat any single number with caution.

Set robots.txt rules per crawler type

Robots.txt is a plain-text file at the site root that tells crawlers which paths they may fetch. It uses one group per user-agent, with Allow and Disallow lines. The matching is by token, so each AI crawler gets its own group.

A practical pattern: allow the retrieval bots you want citations from, and decide separately on training bots. If you want Perplexity citations, leave PerplexityBot allowed. If you don't want your content in OpenAI training data, disallow GPTBot but keep OAI-SearchBot allowed.

Here's a minimal example. It's illustrative, not a recommendation for every site:

• User-agent: GPTBot — Disallow: /
• User-agent: OAI-SearchBot — Allow: /
• User-agent: PerplexityBot — Allow: /
• User-agent: ClaudeBot — Disallow: /
• User-agent: Claude-SearchBot — Allow: /

Two cautions. First, robots.txt is voluntary: a bot that ignores it will still crawl. Second, blocking a retrieval bot removes you from that engine's answer pool, so the cost is visibility, not just privacy.

Four common misconceptions, corrected

Misconception one: "Blocking AI crawlers protects my SEO." It doesn't protect rankings, because Googlebot is separate. It does remove you from AI answers if you block the retrieval bots.

Misconception two: "GPTBot and OAI-SearchBot are the same bot." They're separate tokens with separate jobs, and OpenAI documents them separately. Blocking one does not block the other.

Misconception three: "Google-Extended blocks Googlebot." It doesn't. Google's documentation says Google-Extended controls use of content for Gemini training and grounding, not Google Search crawling or ranking.

Misconception four: "AI visibility is just SEO." Traditional SEO targets a ranked list of links. AI citation targets a generated answer that may quote one or two sources. The retrieval bot has to fetch the page for that quote to happen, so the mechanics differ even when the content overlaps.

Pick your rule from your goal, not from a default

The right rule depends on your goal. If you want maximum AI citation coverage, allow every retrieval bot and treat training bots as a separate policy choice. If you want to keep content out of training data but stay citable, block training tokens only.

If you want neither training nor citation, block both. That's a legitimate choice for paywalled or licensed content, but it means AI engines have no path to quote you.

There's no universal answer. The decision depends on your business model, your content licensing and how much you value AI answer visibility versus training-data control.

Verify the change actually worked

After editing robots.txt, fetch it in a browser and confirm the file is served at the root with a 200 status. Then wait. Crawl schedules vary, and there's no public figure for how quickly each vendor re-reads robots.txt.

Check logs again after two to four weeks. Look for the token you allowed and confirm it now appears. If it doesn't, check that the file isn't blocked by a firewall or a CDN rule, because a WAF block overrides robots.txt.

Test one page directly. Ask the engine a question that page should answer, and see whether it cites you. That's a sample check, not a promise, because answer selection depends on the query and the engine's ranking.

Key Facts at a Glance

• Training crawlers collect data for model building; retrieval crawlers fetch pages at answer time and drive citations.
• GPTBot, ClaudeBot and Google-Extended relate to training or control; OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot relate to retrieval.
• Google-Extended is a robots.txt control token, not a crawler, per Google's documentation.
• Robots.txt matches by user-agent token, so each bot needs its own group.
• Verify hits in server logs and check IP ranges against the vendor's published list.
• Blocking retrieval bots removes you from that engine's answer pool; blocking training bots does not.