Blocking GPTBot won't keep your content out of ChatGPT's answers. That's the myth. GPTBot collects data for model training. OAI-SearchBot fetches live pages for search and citations. Block one and you change a different outcome than you might think. This guide covers the main AI crawler types, corrects the myths, and gives you a decision matrix you can use.
AI crawlers split into two jobs: training and retrieval
Training crawlers collect content to build or improve a model. Retrieval crawlers fetch pages in real time to answer a user's question and cite sources. A training crawler downloads pages to create a dataset for model training. A retrieval crawler fetches a page at the moment a user asks a question, so the answer can include a citation.
Block a retrieval crawler and you cut yourself off from citations in AI answers. Block a training crawler and you opt out of model training, but you may still appear in live answers through the retrieval crawler. The two decisions are separate. Many site owners treat them as one, and that's where things go wrong.
Simple rule: allow retrieval bots if you want citations. Decide on training bots based on your content policy. The rest is detail.
The user-agent string is your first clue, but it's not proof
Each bot identifies itself in the HTTP request header. GPTBot's user-agent contains "GPTBot". OAI-SearchBot's contains "OAI-SearchBot". You can check your server logs to see which one visited. The user-agent is a string of text a bot sends to identify itself. It's not a security measure, because anyone can fake it.
Purpose is the second clue. Training crawlers often crawl broadly and deeply, building a corpus. Retrieval crawlers tend to fetch specific pages that match a query, often with a referrer or query parameter. That's a pattern, not a rule. The only reliable way to know is to check the bot's documentation.
OpenAI documents GPTBot as a crawler for training data, and OAI-SearchBot as a crawler for search results and citations. Perplexity documents PerplexityBot as a crawler for its answer engine. Google documents Google-Extended as a control for how content is used in Gemini training, not as a separate crawler. These are platform-specific behaviours, and they change. Always check the current documentation.
| Bot name | Type | Purpose | Effect on citations | Worth optimising for? |
|---|---|---|---|---|
| GPTBot | Training | Collects data for model training | Blocking it does not directly stop live citations | Only if you want to opt out of training |
| OAI-SearchBot | Retrieval | Fetches pages for search and citations | Blocking it can remove you from ChatGPT search results | Yes, if you want citations |
| PerplexityBot | Retrieval | Fetches pages for Perplexity answers | Blocking it can remove you from Perplexity citations | Yes, if you want citations |
| ClaudeBot | Training | Collects data for Anthropic model training | Blocking it does not directly affect live answers | Only if you want to opt out of training |
| Google-Extended | Control token (not a crawler) | Controls use of content in Gemini training | Does not control crawling or citations | No; it's a policy setting |
The table reflects current documentation as of September 2026. Platform policies can change; check the official docs before you act.
Three mistakes trip up most teams
Mistake one: treating Google-Extended as a crawler. It's not. Google-Extended is a robots.txt token that lets you opt out of Gemini training. It does not control Googlebot or stop your pages from appearing in Google Search. Google's documentation states that Google-Extended does not affect Search. If you block it, you only affect training use.
Mistake two: blocking GPTBot and thinking you've blocked ChatGPT. ChatGPT's search feature uses OAI-SearchBot. If you allow OAI-SearchBot, your pages can still be fetched for citations. Block GPTBot only if you want to opt out of training data collection. The two decisions are independent.
Mistake three: assuming all AI crawlers are the same. They're not. Each bot has its own purpose and its own robots.txt token. A blanket block can hurt your visibility in AI answers. A targeted policy is better.
Pick your goal first, then set robots.txt per bot
Start with your goal. If you want citations in AI answers, allow retrieval bots like OAI-SearchBot and PerplexityBot. If you want to limit training data collection, block training bots like GPTBot and ClaudeBot. You can do both: allow retrieval, block training. That's a common setup.
If you're not sure, allow retrieval bots first. Citations bring traffic and brand visibility. Training is a separate decision. Many publishers allow retrieval and block training. That's a valid choice.
If you have premium content, you might block both. But then you lose AI visibility. There's a trade-off. There's no public figure on how much traffic AI citations bring; it varies by niche. You'll need to measure it yourself.
Use robots.txt to set rules per bot. For example:
- User-agent: GPTBot
Disallow: / - User-agent: OAI-SearchBot
Allow: / - User-agent: PerplexityBot
Allow: /
This allows retrieval but blocks training. You can adjust based on your policy. Always test your robots.txt with a validator. Remember that robots.txt is a voluntary standard; bots may ignore it, but reputable ones follow it.
Verify visits by IP, not just user-agent
Check your server logs. Look for user-agent strings that match known bots. A request with user-agent containing "GPTBot" indicates a visit from GPTBot. But user-agents can be spoofed, so verify by IP address if possible. OpenAI publishes IP ranges for its crawlers. You can match the IP to confirm.
If you use a CDN like Cloudflare, you can use its bot management tools to identify verified bots. Cloudflare maintains a list of verified bots. That's more reliable than user-agent alone.
To do the check: filter your logs for the bot's user-agent, then check the IP against the published range. If they match, it's likely genuine. If not, treat it as suspicious. There's no public figure on how many fake bot requests occur; it depends on your site's popularity.
Bot policy drives citations directly; training affects model knowledge indirectly
Your bot policy affects whether your content can be cited. If you block retrieval bots, you won't appear in AI answers from those platforms. That's a direct link. Training bots don't directly affect citations, but they affect model knowledge over time. The effect is indirect and hard to measure.
To measure AI visibility, track referrals from AI platforms in your analytics. Look for traffic from chatgpt.com, perplexity.ai, and similar. You can also manually query the AI platforms to see if your brand is cited. There's no standard tool for this; it's manual work.
Set a baseline before you change your robots.txt. Then monitor referrals and citations over the next few weeks. Expect changes to appear gradually, not overnight. If you block a retrieval bot, citations from that platform should drop. If you allow it, they may grow. But there's no promise; it depends on many factors.
Key Facts at a Glance
- Training crawlers collect data for model training; retrieval crawlers fetch live pages for citations.
- GPTBot and ClaudeBot are training crawlers; OAI-SearchBot and PerplexityBot are retrieval crawlers.
- Google-Extended is a robots.txt token, not a crawler; it controls Gemini training use only.
- Blocking retrieval bots can remove you from AI citations; blocking training bots does not directly affect citations.
- You can allow retrieval and block training separately using robots.txt user-agent rules.
- Verify bot visits by checking user-agent and IP against published ranges.
Check these before you buy an AI visibility tool
If you're evaluating a tool or service that claims to optimise AI visibility, ask these questions. First, does it respect robots.txt per bot? Second, can it show you which bots actually crawl your site? Third, does it tie changes to measurable citations? Fourth, does it distinguish training from retrieval? Fifth, does it provide a way to verify bot identities? If the answer is no to any of these, be cautious.
Also, check the tool's own documentation for how it handles AI crawlers. Some tools may block all AI bots by default, which could hurt your visibility. Make sure you understand the default settings.
Finally, remember that AI crawler behaviour is not fixed. Platforms update their bots and policies. Review your settings every few months. There's no set-and-forget solution.
What is the difference between a training and retrieval crawler?
A training crawler collects content to build or improve an AI model. A retrieval crawler fetches pages in real time to answer a user's question and cite sources. They serve different purposes and are controlled by different robots.txt tokens.
Should I block GPTBot but allow OAI-SearchBot?
That depends on your goal. If you want to opt out of training data collection but still appear in ChatGPT search results, then yes: block GPTBot and allow OAI-SearchBot. If you want to be in both, allow both. Check OpenAI's current documentation for the latest guidance.
Does blocking PerplexityBot affect my AI search visibility?
Yes. PerplexityBot is a retrieval crawler used by Perplexity to fetch pages for its answers. If you block it, your pages won't be fetched for Perplexity citations. That can reduce your visibility in Perplexity answers.
Is Google-Extended a crawler?
No. Google-Extended is a robots.txt token that lets you control whether your content is used in Gemini training. It does not crawl your site. Googlebot is the crawler for Google Search.
How do I check which AI crawlers visit my site?
Check your server logs for user-agent strings that match known bots. Verify the IP address against the bot's published IP ranges. Tools like Cloudflare's verified bots can help. User-agents can be spoofed, so IP verification is important.
Data sources: the standards and technical parameters referenced in this article are based on the corresponding national/industry standards and the manufacturer's factory QC records; the current in-force version prevails.