Most guides say training crawlers build model data and retrieval crawlers fetch live pages for citations. That split is real. The trouble is that most guides put the wrong bots on each side. Here's the short version: the user-agent name tells you who is asking. The operator's own documentation tells you what happens next. The job isn't built into the name.

This matters because robots.txt blocks work per user-agent, not per purpose. One user-agent can do more than one job. A block aimed at training can quietly pull your pages out of live answers too.

The operator's docs, not the bot's name, tell you which type a crawler is

A training crawler collects pages so a model can learn patterns from them later. A retrieval crawler fetches a page at answer time so a system can quote or summarise it. You get those definitions from what the operators publish, not from the bot's name.

OpenAI documents GPTBot as the crawler that collects content which may be used to train its models. It documents OAI-SearchBot as the crawler behind search results in ChatGPT. It documents ChatGPT-User as the agent that fetches a page when a user asks ChatGPT to open or act on it. One company, three user-agents, three jobs.

Perplexity documents PerplexityBot as the crawler that indexes pages for its search results. That's a retrieval job, since the fetched page feeds an answer. Its documentation also says pages crawled by PerplexityBot may be used to improve its models. So the clean training/retrieval label doesn't fit either.

Check the operator's documentation page for the specific user-agent. Skip the third-party list. A list written in 2024 may be stale by September 2026.

The table sorts each bot by type, purpose, citation effect and whether to optimise

The table below has five columns: user-agent, type, purpose, effect on citations, and whether the bot is worth optimising for. Each row points to the operator's own documentation. Where a platform hasn't published a clear statement, the cell says so.

User-agentTypePurpose (per operator docs)Effect on citationsWorth optimising?
GPTBotTrainingCollects content that may be used to train OpenAI modelsNot a live citation fetch; affects future model knowledgeYes, if you want model knowledge to include your pages. Blocking removes that path.
OAI-SearchBotRetrieval / search indexPowers search results shown in ChatGPTDirect: pages it can read can appear in ChatGPT search answersYes, this is the one to allow if you want ChatGPT visibility
ChatGPT-UserRetrieval / user-initiatedFetches a page when a user asks ChatGPT to open or act on itDirect: the page the user asked forYes, if you want users to reach your page through ChatGPT
PerplexityBotRetrieval / search index, with model improvementIndexes pages for Perplexity search; may improve modelsDirect: indexed pages can be cited in Perplexity answersYes, for Perplexity visibility
Perplexity-UserRetrieval / user-initiatedFetches a page on a user's requestDirect: the page the user asked forYes, if you want users to reach your page through Perplexity
GooglebotSearch index (not AI-specific)Google Search indexingIndirect: feeds Google Search, and Google's AI features draw on the same indexYes, always
Google-ExtendedTraining / AI use controlControls whether content is used for Gemini training and grounding, per Google's documentationNot a fetch; a control token on Googlebot's crawlDepends on your AI policy; blocking does not remove you from Google Search

Google-Extended is a control token, not a crawler. It changes how Google may use content that Googlebot already crawled. Treating it as a separate bot is one of the common errors in audit reports.

Three misconceptions cause most of the bad advice

Misconception one: "GPTBot is how ChatGPT cites pages." It isn't. OpenAI's documentation puts GPTBot on the training side and OAI-SearchBot on the search side. Blocking GPTBot alone doesn't remove you from ChatGPT search answers. Allowing GPTBot alone doesn't put you in them.

Misconception two: "PerplexityBot is a training crawler." Its documented job is indexing for search results. The model-improvement clause is a secondary use, not the main one. Call it a training crawler and block it, and you drop out of Perplexity answers.

Misconception three: "robots.txt is a legal block." It's a voluntary protocol. RFC 9309, the Robots Exclusion Protocol standard published in 2022, defines how compliant crawlers should read robots.txt. A crawler that ignores the file isn't breaking a law. It's breaking a convention. Real access control needs authentication or a firewall rule.

Each of those errors leads to a block that doesn't do what the writer intended.

Set robots.txt by goal: allow retrieval, block training

Decide the goal first. If you want your pages in AI answers, the retrieval user-agents need to be allowed. If you don't want your pages used to train future models, the training user-agents need to be blocked. Those are two separate decisions, and robots.txt handles them with separate rules.

A retrieval-friendly, training-restricted setup looks like this: block GPTBot, allow OAI-SearchBot, allow ChatGPT-User, allow PerplexityBot, allow Perplexity-User. Whether you also block Google-Extended depends on your stance on Gemini training. It doesn't affect Google Search ranking.

A fully closed setup blocks all of the above. That's a valid choice for paywalled or licensed content, but make it a deliberate one. Check the log after a month to confirm the blocks are taking effect. The user-agent string in the log tells you which bot actually hit the server.

One more trap: robots.txt rules match by user-agent token, not by company. A rule for "GPTBot" doesn't cover "OAI-SearchBot". Write each token explicitly. There's no wildcard that means "all OpenAI bots".

Check a crawler's identity with DNS, not the user-agent string

User-agent strings are easy to fake. A scraper can send "GPTBot" as its user-agent and pretend to be OpenAI. The check that works is a reverse DNS lookup, then a forward lookup on the resulting hostname, to confirm the IP belongs to the operator's published range.

OpenAI publishes its crawler IP ranges in a public file. Google publishes its ranges in a similar JSON file. Perplexity publishes its IP ranges on its documentation site. Compare the connecting IP against those lists before you allow or block on the user-agent alone.

For a first-pass audit, pull the server log and group hits by user-agent. Then run the DNS check on a sample of the IPs. If a "GPTBot" hit comes from an IP outside OpenAI's range, it isn't GPTBot, whatever the user-agent says.

Audit which AI crawlers actually reach your site

Start with the raw access log. Filter for known AI user-agent tokens: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended (as a control reference), ClaudeBot, and CCBot. Count hits per token per week. That gives you a baseline.

Next, cross-check the counts against your robots.txt rules. If a token is blocked but still shows hits, either the crawler is ignoring the file or the hits are spoofed. Both are worth knowing.

Then look at what those bots actually fetched. A retrieval crawler tends to hit the pages a user asked about, which may include deep pages and query strings. A training crawler tends to sweep broadly. The pattern is a clue, not proof. The operator's documentation is the proof.

Repeat the audit monthly. Crawler behaviour and documentation change, and a rule that was right in March can be wrong by September.

Optimise for retrieval, because training gives you little to do

Optimising for a retrieval crawler means making pages easy to fetch and easy to quote: clear headings, a direct answer near the top, stable URLs, and no JavaScript-only content that the bot can't render. Optimising for a training crawler means almost nothing on the page side. The bot takes what it takes.

That asymmetry is the practical reason the distinction matters. If you want citations, your work goes into retrieval-facing pages. If you only care about future model knowledge, there's little you can do beyond allowing access.

One caveat: allowing retrieval doesn't ensure a citation. It only keeps the door open. Whether a system quotes your page depends on its own ranking and selection logic, which the operators don't fully publish.

Key Facts at a Glance

  • Training crawler: collects pages for model learning. Retrieval crawler: fetches a page at answer time. The user-agent name doesn't tell you which.
  • GPTBot is documented as training; OAI-SearchBot and ChatGPT-User are documented as retrieval. Blocking GPTBot alone doesn't remove you from ChatGPT search.
  • PerplexityBot is documented as a search indexer with a model-improvement clause, not a pure training crawler.
  • Google-Extended is a control token, not a crawler. It doesn't affect Google Search inclusion.
  • robots.txt is a voluntary protocol under RFC 9309. For real access control, use authentication or firewall rules.
  • Verify crawler identity with reverse DNS against the operator's published IP ranges, not the user-agent string alone.
What is the difference between a training crawler and a retrieval crawler?

A training crawler collects pages so a model can learn patterns from them later. A retrieval crawler fetches a page at answer time so a system can quote or summarise it. The same company can run both, under different user-agent names.

How do I block GPTBot but allow PerplexityBot?

Write two separate user-agent groups in robots.txt. One group for GPTBot with Disallow: /. One group for PerplexityBot with Allow: /. Then check the log after a month to confirm each rule is being read as intended. Remember that robots.txt is voluntary and doesn't stop a crawler that ignores it.

Does blocking AI crawlers hurt my SEO?

Blocking GPTBot doesn't affect Google Search, because Googlebot is a separate crawler. Blocking Google-Extended doesn't remove you from Google Search either; it only limits use for Gemini training and grounding. Blocking OAI-SearchBot or PerplexityBot can reduce your visibility in those AI answers, which is a separate channel from classic SEO.

How can I verify an AI crawler is legitimate?

Do a reverse DNS lookup on the connecting IP, then a forward lookup on the resulting hostname, and compare against the operator's published IP ranges. OpenAI, Google and Perplexity all publish their ranges. A user-agent string on its own proves nothing.

Which AI crawlers should I allow in robots.txt?

It depends on your goal. For AI answer visibility, allow the retrieval user-agents: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User. For training control, block the training user-agents: GPTBot, and Google-Extended if you want to limit Gemini use. There's no single right answer; the choice is a business decision, not a technical default.