Most teams treat AI crawlers as one group. They see a bot in the logs, block it, and assume nothing changes. The correction is simple: training crawlers and retrieval crawlers do different jobs, and blocking one is not the same as blocking the other. A training crawler collects pages to build a model later. A retrieval crawler fetches pages to answer a question now, and that answer may carry a link back to your site.

A training crawler builds a model; a retrieval crawler answers a question

A crawler is a program that fetches web pages automatically by following links or a list of URLs. The purpose behind the fetch is what separates the two types. Training crawlers gather content for model development. Retrieval crawlers fetch a page at answer time, often because a user just asked a question.

The timing matters more than the user-agent name. Training collection happens in bulk and on the operator's schedule. Retrieval happens on demand, so a single page might be fetched seconds before someone reads a summary of it. Check the request pattern in the logs before deciding which type hit the site.

This is not a clean split. Some operators run both kinds of crawler under separate user-agent strings, and some run one crawler that serves several products. The label on the bot is a starting point, not proof of intent.

Each AI operator publishes its own crawler tokens, and they do different jobs

OpenAI documents GPTBot as its crawler for collecting content that may be used to train its models. OpenAI also documents OAI-SearchBot for search results and ChatGPT-User for user-triggered visits. Those are three different jobs under one company, and they are separate entries in robots.txt.

Perplexity documents PerplexityBot as the crawler behind its search index. Perplexity states that pages surfaced in answers link back to the source. Anthropic documents ClaudeBot for model training and Claude-User for user-initiated fetches. Google documents Google-Extended as a control for how its content is used in Gemini training and grounding, separate from Googlebot, which handles Search.

Here is the practical problem. A robots.txt rule aimed at "GPT" or "AI" matches nothing, because robots.txt matches user-agent tokens, not themes. If you want to allow or block a named crawler, you need its exact token.

The documented crawlers split cleanly into training and retrieval roles

The table below maps each documented crawler to its type, its purpose, and whether tuning your site for it is worth the effort. User-agent tokens and purposes come from each operator's own documentation. Check the current version of that documentation before you write rules, because operators rename and add crawlers.

Crawler (user-agent token)TypePurposeEffect on citationsWorth optimizing?
GPTBot (OpenAI)TrainingCollects content that may be used to train OpenAI modelsIndirect. It does not fetch pages to answer a live query.Low for citation, moderate for long-term model exposure
OAI-SearchBot (OpenAI)Retrieval / searchBuilds and maintains OpenAI's search indexDirect. Indexed pages can appear in search results with links.High
ChatGPT-User (OpenAI)Retrieval, user-triggeredFetches a page when a user asks ChatGPT to visit itDirect for that sessionHigh
PerplexityBot (Perplexity)Retrieval / search indexCrawls pages for the Perplexity search indexDirect. Answers cite and link to sources.High
ClaudeBot (Anthropic)TrainingCollects content for model trainingIndirectLow for citation
Claude-User (Anthropic)Retrieval, user-triggeredFetches pages on behalf of a user requestDirect for that sessionHigh
Google-Extended (Google)Training / grounding controlControls use of content in Gemini training and grounding; does not affect Search inclusionIndirectDepends on your Gemini strategy
Googlebot (Google)Search indexingCrawls pages for Google SearchIndirect but foundationalEssential

One caveat on the table: it reflects published documentation at the time of writing. Treat "type" as the operator's stated purpose, not as a promise about how every request is used.

Blanket blocks and CDN defaults are the two mistakes that cost citations

The most common mistake is a blanket rule. A line like User-agent: * followed by Disallow: / blocks every crawler that follows robots.txt, including the retrieval crawlers you wanted. Teams add that rule to stop one scraper and quietly cut off their own citation path.

The second mistake sits one layer down, in the CDN or WAF. Bot-management features often ship with defaults that challenge or block automated traffic. Cloudflare documents its bot management and its own crawler controls, and the defaults are not built around your AI citation goals. If a retrieval crawler gets a challenge page instead of your HTML, it has nothing to cite.

The third mistake is user-agent matching. Robots.txt matching is prefix-based on the token, so a rule for PerplexityBot does not cover Perplexity-User. Write each token on its own line and test the file before you ship it. The robots.txt specification (RFC 9309) defines how user-agent tokens are matched.

Raw access logs show what actually reached your server

Start with the raw access log, not an analytics dashboard. Analytics tools filter bot traffic by default, so the crawler you are looking for may never appear in the report. The log line shows the user-agent string, the requested URL, the status code, and the source IP.

Match the user-agent string against the operator's published token. Then check the status code. A 200 means the page was served. A 403 or a 429 means something blocked or throttled the request, and that is the line worth chasing.

If your logs are thin, verify the crawler against the operator's published IP ranges where they exist, and check whether your CDN logs requests separately from the origin. A retrieval crawler that gets blocked at the edge never shows up in origin logs at all.

Decide access by content type, not by opinion about AI

Skip the debate about AI in general and map the decision to content. Public pages that you want cited should stay open to retrieval crawlers. Paywalled, licensed, or internal content is a different case, and a targeted block is reasonable there.

  • Public product and help pages: allow retrieval crawlers, because citations send qualified traffic.
  • Research and data pages: allow retrieval, decide separately on training based on your licensing position.
  • Paywalled or subscriber content: block both types unless you have a licensing deal.
  • Staging and admin paths: block everything, and check that the block is not catching production.

Training access is a business call, not a technical one. Blocking training crawlers does not remove pages you already published from a model, and allowing them does not ensure inclusion. There is no public figure for how much a single page contributes to a model's output.

llms.txt and structured data only help after access is open

An llms.txt file is a proposed plain-text file at the site root that gives AI systems a curated list of key pages. It is a proposal, not a ratified standard, and adoption varies by operator. Treat it as a supplement to crawlable HTML, not a replacement.

Structured data does the heavier lifting. Schema.org defines the vocabulary, and search and answer systems read it to understand entities and relationships on the page. A retrieval crawler that can parse clean markup has an easier job than one parsing a wall of unstructured text.

None of this overrides robots.txt. If a retrieval crawler is blocked at the robots level, the llms.txt file and the markup never get read. Fix access first, then improve the content.

The Short Version

  • Training crawlers collect content for model building. Retrieval crawlers fetch pages to answer a live question.
  • GPTBot and ClaudeBot are documented as training crawlers. PerplexityBot and OAI-SearchBot are documented as search or retrieval crawlers.
  • Robots.txt matches exact user-agent tokens. A rule for "AI bots" matches nothing.
  • CDN and WAF bot defaults are a common cause of silent blocking. Check them before you blame the crawler.
  • Check raw server logs and status codes, not filtered analytics reports.
  • Allow retrieval crawlers on public pages you want cited. Decide on training access as a separate business question.

Key Facts at a Glance

  • Training crawlers (GPTBot, ClaudeBot) collect content for model development; retrieval crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-User) fetch pages to answer live queries.
  • Robots.txt matches user-agent tokens, not themes; a rule for "AI" matches nothing.
  • RFC 9309 defines robots.txt matching; prefix-based rules mean PerplexityBot does not cover Perplexity-User.
  • Cloudflare bot management defaults can challenge or block retrieval crawlers before they reach your origin.
  • Google-Extended controls Gemini training and grounding but does not affect Google Search inclusion.
What is the difference between training and retrieval crawlers?

A training crawler collects pages to build or improve a model, usually in bulk and on the operator's schedule. A retrieval crawler fetches a page at answer time, often because a user asked a question, and that fetch can lead to a citation.

How is GPTBot different from PerplexityBot?

OpenAI documents GPTBot as its training crawler. Perplexity documents PerplexityBot as the crawler for its search index, which surfaces and links to sources in answers. They are separate robots.txt tokens with separate purposes.

Should I block AI crawlers in robots.txt?

It depends on the content and the crawler type. Public pages you want cited should stay open to retrieval crawlers. Paywalled or licensed content is a reasonable case for a targeted block. Decide training access separately from retrieval access.

How do I check if AI crawlers are visiting my site?

Read the raw server access log and match the user-agent string against the operator's published token. Check the status code too, because a 403 or 429 shows the request was blocked or throttled.

Does blocking AI crawlers affect my SEO?

Blocking Google-Extended does not change Google Search inclusion, because Google documents it as a separate control for Gemini training and grounding. Blocking Googlebot does affect Search. For other operators, the effect depends on which crawler you block.

Data sources: OpenAI, Perplexity, Anthropic, Google, Cloudflare documentation, and RFC 9309 (robots.txt specification).