Training crawlers collect data to build or improve AI models. Retrieval crawlers fetch pages at the moment a user asks a question, so an AI can quote them in the answer. That's the core split, and it changes how you treat each bot in robots.txt.
One feeds the model before the question exists. The other feeds the answer while the question is live. Block the wrong one and you can lose citations without protecting anything.
How to Tell a Training Crawler from a Retrieval Crawler
A training crawler gathers web content for model training. The data gets stored and used later, often months before you see any result. A retrieval crawler fetches a page in real time to answer a specific query. It doesn't keep the page for training; it reads it now, cites it now, and moves on.
The two have different goals. Training crawlers want volume and variety. Retrieval crawlers want relevance and freshness. A page that ranks poorly in search can still be cited by a retrieval crawler if it matches the question well.
The lines blur. Some systems do both. OpenAI's documentation lists GPTBot as a training crawler and OAI-SearchBot as a search crawler that surfaces content in ChatGPT search results. Perplexity's documentation describes PerplexityBot as a search crawler that fetches pages to answer user queries in real time. So the same company can run two bots with two different jobs.
How to Compare the Main AI Crawlers Side by Side
Here's a practical table. The "type" column shows whether the bot is mainly for training, retrieval, or both. The "impact on citations" column explains what happens if you allow it. The "worth optimizing?" column gives a plain answer for most B2B sites.
User-agent tokens and behaviors can change. Always check the vendor's official documentation for the current list.
| Crawler | Owner | Type | Purpose | Impact on citations | Worth optimizing? |
|---|---|---|---|---|---|
| GPTBot | OpenAI | Training | Collects data to train and improve OpenAI models. | No direct citation today; the data may influence future model behavior. | Only if you want to feed future models. Blocking won't hurt current citations. |
| OAI-SearchBot | OpenAI | Retrieval | Fetches pages to show in ChatGPT search results. | Direct impact: your page can be cited in answers. | Yes. Keep it allowed if you want ChatGPT citations. |
| ChatGPT-User | OpenAI | Retrieval (user-triggered) | Fetches a page when a user asks ChatGPT to visit a URL. | Direct impact: the user sees your page in the response. | Yes. Allow it for user-driven visits. |
| ClaudeBot | Anthropic | Training | Collects data to train Claude models. | No direct citation today; may affect future model knowledge. | Only if you want to contribute to training. Blocking won't remove you from current answers. |
| PerplexityBot | Perplexity | Retrieval | Fetches pages in real time to answer user queries. | Direct impact: your page can appear as a source in Perplexity answers. | Yes. Keep it allowed if you want Perplexity citations. |
| Google-Extended | Training (opt-out token) | Controls whether your site feeds Gemini model training. | No direct impact on Search citations. It's separate from Googlebot. | Only if you want to opt out of Gemini training. It doesn't affect Google Search. | |
| Googlebot | Search index | Fetches pages for Google Search indexing. | Direct impact on search rankings and AI Overviews. | Yes. Always allow it. | |
| Bingbot | Microsoft | Search index | Fetches pages for Bing search and Copilot. | Direct impact on Bing rankings and Copilot citations. | Yes. Always allow it. |
Notice the pattern: retrieval crawlers and search index crawlers are the ones that affect what users see today. Training crawlers affect what models learn for tomorrow. If your goal is citations, focus on the retrieval and index bots first.
How to Configure robots.txt for Different Goals
Robots.txt is a plain text file at the root of your domain. It tells crawlers which paths they may or may not fetch. It's a request, not a law. Well-behaved bots follow it; others may ignore it.
Here's what a typical configuration looks like if you want to allow retrieval crawlers but block training crawlers:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
This setup blocks OpenAI's training crawler and Anthropic's training crawler. It allows OpenAI's search crawler, Perplexity's retrieval crawler, and the main search index bots. It also opts out of Google's Gemini training via Google-Extended.
If you want to allow everything, you can simply omit the file or use a wildcard allow. If you want to block all AI crawlers, you'd list each user-agent with a disallow. There's no single "block all AI" token that covers every bot.
One more thing: robots.txt doesn't control caching or snippet display. Those are handled by meta tags like noarchive or nosnippet. And it doesn't stop a bot that ignores the file. For that, you need server-level blocking or a web application firewall.
How to Verify a Crawler's Identity
Anyone can set a user-agent string to "GPTBot." So user-agent alone isn't proof. The reliable method is reverse DNS lookup. You take the crawler's IP address, look up its hostname, and check that the hostname resolves back to the same IP.
OpenAI publishes a list of IP ranges for its crawlers in its documentation. Anthropic and Perplexity do the same. If a request claims to be from one of these bots but its IP doesn't match the published range, it's probably not genuine.
Google's documentation on verifying Googlebot describes the same reverse DNS method. The principle is the same across vendors: check the IP, not just the name.
For log analysis, you'd filter requests by user-agent, then run reverse DNS on the source IPs. If you see a lot of "GPTBot" requests from IPs outside OpenAI's range, that's a sign of spoofing. You can then block those IPs at the firewall.
How Blocking Affects AI Visibility
Blocking a training crawler doesn't remove your site from current AI answers. It only stops your content from being used in future training runs. So if you block GPTBot today, ChatGPT can still cite you tomorrow via its search crawler or via user-triggered fetches.
Blocking a retrieval crawler does affect current citations. If you block PerplexityBot, Perplexity can't fetch your page to answer a user's question. You won't appear as a source in those answers. The same goes for OAI-SearchBot and ChatGPT search.
There's no public figure for how much traffic or citations you lose by blocking a specific bot. It depends on your niche, your content, and how often users ask questions that match your pages.
The safe default for most B2B sites: allow retrieval crawlers and search index bots. Decide on training crawlers based on your content strategy. If you want your expertise to shape future models, allow them. If you'd rather keep your content out of training data, block them.
How to Avoid Common Misconceptions
One common mistake is thinking that blocking GPTBot also blocks ChatGPT from citing you. It doesn't. ChatGPT uses different crawlers for search and user-triggered visits. Blocking the training bot only stops training use.
Another mistake is assuming Google-Extended affects Google Search. It doesn't. Google-Extended is a standalone token that controls Gemini training and grounding. Google Search uses Googlebot, which is separate. Blocking Google-Extended won't change your rankings.
A third mistake is treating robots.txt as a security tool. It's not. It's a convention. Bots that respect it will obey; bots that don't will ignore it. If you need enforcement, use server-side blocks or a WAF.
Finally, some people think all AI crawlers are the same. They're not. Training crawlers, retrieval crawlers, and search index crawlers have different jobs and different impacts. Treating them as one group leads to bad decisions.
The Short Version
- Training crawlers (GPTBot, ClaudeBot) collect data for future models. Blocking them doesn't affect current citations.
- Retrieval crawlers (PerplexityBot, OAI-SearchBot) fetch pages in real time to answer questions. Blocking them removes you from those answers.
- Search index crawlers (Googlebot, Bingbot) feed search results and AI Overviews. Always allow them.
- Google-Extended is a separate token for Gemini training. It doesn't affect Google Search.
- Verify crawler identity with reverse DNS, not just user-agent strings.
- Robots.txt is a request, not a security measure. Use server rules for enforcement.
Key Facts at a Glance
- Training crawlers and retrieval crawlers serve different purposes: one builds models, the other answers live queries.
- GPTBot is a training crawler; OAI-SearchBot and ChatGPT-User are retrieval crawlers.
- PerplexityBot is a retrieval crawler that fetches pages in real time for answers.
- Blocking a training crawler doesn't remove you from current AI answers; blocking a retrieval crawler does.
- Google-Extended controls Gemini training, not Google Search.
- Use reverse DNS to confirm a crawler's identity; user-agent strings alone can be spoofed.
What is the difference between training and retrieval crawlers?
Training crawlers collect data to build or improve AI models. Retrieval crawlers fetch pages in real time to answer a specific user question. Training data is stored for later; retrieval data is used now.
How do I block GPTBot from my site?
Add a rule to your robots.txt file: User-agent: GPTBot followed by Disallow: /. This asks GPTBot not to crawl your site. It doesn't affect other OpenAI crawlers like OAI-SearchBot.
Does blocking PerplexityBot affect my SEO?
It doesn't affect Google or Bing rankings. It does stop Perplexity from fetching your pages to answer user queries, so you won't appear as a source in Perplexity answers.
Can AI crawlers be identified in server logs?
Yes, but user-agent strings can be spoofed. The reliable method is reverse DNS lookup: check that the crawler's IP resolves to a hostname that matches the vendor's published range.
What happens if I allow all AI crawlers?
Your content can be used for training and for live answers. You may see more citations in AI tools, but you also contribute your content to future models. The trade-off depends on your content strategy.