A supplier ships a batch of coated glass. Weeks later a competitor's AI answer quotes the spec sheet word for word. The supplier never blocked the crawler. They never allowed it on purpose either. That's the usual story.

Here's the short answer. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and OAI-SearchBot are separate user-agents. A rule for one does not touch the others. Decide on each one by business goal, not with a single blanket line.

A robots.txt rule for AI crawlers is a line that tells one named bot whether it may fetch your pages. The file lives at the root of your domain, /robots.txt. The IETF RFC 9309 (2022) describes how crawlers should read it. It's a voluntary exclusion protocol, not access control. A bot that ignores the file will still fetch your pages.

A token is the user-agent name a crawler sends, and each one belongs to a different owner

You match that name in robots.txt to give the bot rules. The table below lists the main AI-related tokens, who owns them, and what the owner says the token does.

TokenOwnerWhat the owner says it doesBlocking it affects
GPTBotOpenAIFetches pages for model training, per OpenAI's GPTBot documentationTraining use of your pages, not ChatGPT's live browsing
OAI-SearchBotOpenAIFetches pages for ChatGPT search results, per OpenAI's crawler docsWhether your pages can appear in ChatGPT search answers
ChatGPT-UserOpenAIFetches a page when a user asks ChatGPT to open it, per OpenAI docsOn-demand visits from user prompts
ClaudeBotAnthropicFetches pages for model training, per Anthropic's crawler support pageTraining use of your pages
Claude-UserAnthropicFetches pages when a user asks Claude to visit one, per AnthropicOn-demand visits from user prompts
PerplexityBotPerplexityFetches pages to build Perplexity's search index, per Perplexity's bot documentationWhether your pages can be cited in Perplexity answers
Google-ExtendedGoogleA control token for Gemini training and grounding, per Google's crawler docs; it does not change Googlebot's search crawlingGemini training and grounding use, not Google Search ranking
GooglebotGoogleFetches pages for Google Search indexing, per Google Search CentralYour normal Google Search presence
CCBotCommon CrawlFetches pages for the public Common Crawl dataset, per Common Crawl's siteThat dataset, which many model builders reuse

Two things stand out. Training bots and answer bots are different products. GPTBot and ClaudeBot feed model training. OAI-SearchBot and PerplexityBot feed live answers. Blocking the first does not block the second.

Google-Extended is not Googlebot. Google's own documentation says Google-Extended lets site owners manage whether their content is used for Gemini training and grounding. It does not affect how Googlebot crawls for Search.

Robots.txt only controls the fetch, so a block stops the page before anything else happens

Picture the supplier's spec sheet. A crawler fetches it, a parser reads the text, and the content lands in an index or a training set. When a user asks a question, a retrieval step pulls candidate pages, and the answer engine cites one or two.

Robots.txt sits at the very first step, the fetch. If the rule blocks the bot, the later steps never happen for that bot. If the rule allows it, the page can move forward. Nothing in robots.txt ensures a citation.

That's why blocking has trade-offs. A blocked training bot keeps your text out of that model's training data. A blocked answer bot keeps your page out of that engine's citations. Those are separate decisions with separate costs.

Put simply: robots.txt is a door policy, not a ranking system. It decides who gets in. The rest is up to the engine's own relevance and quality checks.

Robots.txt groups start with a User-agent line, and one stray wildcard can block every bot

A group starts with a User-agent line, and Disallow or Allow lines follow. A Disallow: / blocks the whole site for that bot. A Disallow: with nothing after the colon blocks nothing.

The snippet below is a starting point. Edit the token list to match your decision, and test it before you publish.

# Allow answer bots, block training bots
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /

One trap: a group has no effect unless it names the bot. Writing User-agent: * with Disallow: / blocks every bot that follows the rules and is not named elsewhere. Many sites add that line during a staging phase and forget it.

Another trap: Allow and Disallow order does not matter for most crawlers, but the most specific matching path wins under RFC 9309. Keep rules short and avoid overlapping paths where you can.

Allow or block by business goal, because there's no single right rule

The decision depends on what you want from AI systems: citations, training exclusion, or both. The matrix below maps common goals to a starting rule set. Treat it as a starting point, not a fixed policy.

Business goalAnswer bots (OAI-SearchBot, PerplexityBot)Training bots (GPTBot, ClaudeBot, CCBot)Google-Extended
Maximise AI citationsAllowAllowAllow
Citations yes, training noAllowDisallowDisallow
Paid or gated contentAllow public pages onlyDisallowDisallow
Full opt-outDisallowDisallowDisallow

Check the logs before choosing a row. If a bot never visits, a block costs you nothing today. If it visits your product pages daily, a block removes a channel you were already using.

There's no public figure for how much AI citation traffic converts, so don't assume one. Measure your own referral and brand-search data instead.

Content-signal states a preference in robots.txt, but it is not part of RFC 9309

Content-signal is a proposed extension that adds preference statements to robots.txt, such as whether content may be used for AI training. A group of publishers put it forward, and it is documented at contentsignals.org.

It is not part of RFC 9309, and crawler support varies. Treat it as a signal of intent, not an enforcement tool. A bot that ignores it will still fetch your pages.

Robots.txt and content-signal can sit in the same file. Robots.txt decides access. Content-signal states preference. Neither one replaces a firewall or a licence agreement.

Logs and IP ranges tell you what crawlers actually do, not what your file says

Robots.txt alone tells you nothing about reality. To check, open your server or CDN logs and search for the token names. Look for the user-agent string and the requesting IP range.

OpenAI, Anthropic and Google each publish their crawler IP ranges in their documentation. Match the IP against the published range, because a user-agent string can be copied by anyone.

Here's a short self-check list:

  • Fetch https://yourdomain.com/robots.txt in a browser and confirm it returns 200, not a redirect or a 404.
  • Search logs for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended over the last 30 days.
  • Confirm each hit's IP sits inside the owner's published range.
  • Check your CDN or WAF rules, because a firewall block overrides robots.txt.
  • Re-check after any site migration or CDN change.

If a bot appears in logs but never cites you, the problem is further down the chain: content quality, structure or retrieval. That's a separate job from robots.txt.

Most teams make the same four mistakes, and a broken robots.txt file is the worst one

The first is blocking everything with User-agent: * and forgetting it during a launch. The second is assuming Google-Extended controls Google Search; it doesn't.

The third is blocking GPTBot and expecting ChatGPT search to stop citing the site. That depends on OAI-SearchBot, a different token. The fourth is writing rules in a subfolder or a staging domain and expecting them to apply to production.

One more: a robots.txt file that returns a 500 error. Under RFC 9309, a server error means crawlers should assume the site is fully disallowed until the file loads again. Check uptime for that file like any other critical page.

Key Facts at a Glance

  • RFC 9309 (2022) is the current robots.txt standard; the current in-force version prevails.
  • GPTBot, ClaudeBot and CCBot are training-related tokens; OAI-SearchBot and PerplexityBot are answer-related.
  • Google-Extended does not change Google Search indexing or ranking.
  • Content-signal is a proposed preference extension, not an RFC 9309 rule.
  • A robots.txt server error can lead compliant crawlers to treat the site as fully disallowed.
Is ChatGPT a web crawler?

No. ChatGPT is the product. OpenAI runs separate crawlers, including GPTBot for training and OAI-SearchBot for search, each with its own token and rules.

Does blocking GPTBot affect ChatGPT search results?

Not directly. ChatGPT search uses OAI-SearchBot, per OpenAI's documentation. Blocking GPTBot affects training use, not the search crawler.

What does Google-Extended actually control?

It controls whether your content is used for Gemini training and grounding, per Google's crawler documentation. It does not affect Googlebot's crawling for Search.

How do I check if AI crawlers can access my site?

Fetch your robots.txt, confirm it returns 200, then search server or CDN logs for each token and match the IP against the owner's published range.

Should I allow or block AI crawlers?

It depends on your goal. Allow answer bots if you want citations. Block training bots if you want to limit training use. Test in logs before committing.