A supplier in Spain opens its server logs and sees a bot called GPTBot asking for product pages. The owner wants AI answers to mention the business but does not want the catalogue used as training material. The answer sits in robots.txt: one user-agent token per bot, an Allow or Disallow rule, and a content-signal line. Each token names a different crawler with a different job, so allowing one does not allow the others.

Robots.txt is a plain-text file at the site root that tells crawlers which paths they may fetch. It is a request, not a lock: the IETF specification RFC 9309 (2022) calls it a voluntary convention, and a crawler that ignores it can still fetch the page. The token is the part that decides which crawler hears the request.

What does a robots.txt token actually control?

A token is the string after User-agent:. It matches one crawler's product name, so a block applies to that crawler only. RFC 9309 says matching is case-insensitive and that the most specific matching group wins.

The file does not switch a bot on or off across the web. It sets rules for one host. A Disallow line stops that crawler from fetching a path on your domain; it says nothing about how the crawler behaves elsewhere.

One common mistake: writing User-agent: GPTBot in the middle of a group meant for *. Groups are separated by each new User-agent line, so a misplaced token can leave the general rule covering paths you meant to open.

Which token belongs to which job: training or search?

This is the split that matters most. Some tokens cover crawlers that collect text for model training; others cover crawlers that fetch pages so an AI answer can quote or link them at the moment someone asks.

OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler behind ChatGPT search results. Anthropic documents ClaudeBot for model training and Claude-SearchBot for search. Perplexity documents PerplexityBot for its answer engine.

Google is the clearest case. Googlebot crawls for Search; Google-Extended is a control token that lets a site owner opt out of Gemini training and grounding without touching Search. Google's own documentation says Google-Extended does not affect Search ranking or snippets.

TokenOwnerJob (per owner's documentation)Blocking it typically affects
GPTBotOpenAITraining crawlUse of your pages in OpenAI model training
OAI-SearchBotOpenAISearch crawl for ChatGPTWhether ChatGPT search can fetch and cite your pages
ClaudeBotAnthropicTraining crawlUse of your pages in Anthropic model training
Claude-SearchBotAnthropicSearch crawlWhether Claude search can fetch your pages
PerplexityBotPerplexityAnswer-engine crawlWhether Perplexity can fetch and cite your pages
GooglebotGoogleSearch crawl and indexingGoogle Search visibility, ranking and snippets
Google-ExtendedGoogleControl token for Gemini training and groundingGemini training use; not Search ranking or snippets
CCBotCommon CrawlOpen web archive used by many model buildersWhether your pages enter the Common Crawl dataset
BytespiderByteDanceCrawler tied to ByteDance productsFetching by that crawler; check current docs

How does one page travel from crawl to citation?

Follow a single product page. A search crawler fetches the URL, reads the HTML, and stores a copy in an index. When a person asks a question, the engine retrieves candidate passages and the answer model may quote one and link the source.

Robots.txt acts at the first step only. RFC 9309 defines it as a rule about fetching, not about indexing or quoting. If a page is disallowed, the crawler should not fetch it, so it has nothing to index or cite from that URL.

This is why blocking a search token removes the page from that engine's answer pool, while blocking a training token leaves the page available for live retrieval. The two decisions are separate, and the token list above is the switchboard.

What does the content-signal line add?

Content-Signal is a directive added to robots.txt that states how a site owner wants content used, beyond just who may fetch it. It is published as a draft proposal by Cloudflare, not an IETF standard, so treat it as a request that participating services may honour.

The draft defines three values: search for including content in search results, ai-train for training models, and ai-input for using content as live input to an AI answer. A site can allow search, refuse training and allow live input in one line.

Example: Content-Signal: search=yes, ai-train=no, ai-input=yes. Placement is at the top of the file, before the first User-agent group. Adoption is uneven, and the draft may change, so pair it with explicit per-bot rules rather than relying on it alone.

How do I write the rules for my own case?

Start from the outcome you want, then pick tokens. There is no universal right answer: a publisher selling subscriptions and a shop that wants to be quoted in AI answers need opposite files.

Block training, allow search. This keeps pages eligible for live citation while refusing training use.

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /

Block everything from AI, keep classic search. Keep Googlebot and Bingbot open, close the AI tokens, and remember that a general * rule does not override a named group.

User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: *
Allow: /

Close private paths for everyone. Account, cart and internal search pages should be closed for all tokens, not just AI ones.

User-agent: *
Disallow: /account/
Disallow: /cart/
Disallow: /search?

Put the most specific group first for readability. Test the file before publishing: Google Search Console has a robots.txt report, and Bing Webmaster Tools has its own tester. A test fetch of the live URL is the only way to confirm the file is served, not blocked by a firewall.

What should I check before publishing the file?

Use this list after any edit. Each item maps to a rule you can verify, not a feeling.

  • The file returns HTTP 200 at /robots.txt and is plain text.
  • Every named group starts with its own User-agent: line.
  • No named token sits inside a * block by accident.
  • Search tokens (Googlebot, Bingbot) are open if you want search visibility.
  • Training tokens match your stated policy on model training.
  • Search-answer tokens (OAI-SearchBot, PerplexityBot, Claude-SearchBot) match your citation goal.
  • Content-Signal, if used, sits at the top and uses the draft's values.
  • A test tool shows the intended result for one allow case and one block case.
  • Server logs show the named bots fetching or being refused as expected.

Why does the token list keep changing?

New AI products bring new crawlers, and owners rename or retire tokens. A file written in 2024 may miss a bot that launched in 2026. There is no central register that every vendor must join.

Practical habit: check the major owners' crawler documentation pages once a quarter, and add any new token to your file in the same pass. Keep the file short and commented, so the next editor sees why each group exists.

If a crawler ignores your rules, robots.txt cannot stop it. Enforcement happens at the server or edge, through a firewall rule or a bot-management setting. Treat robots.txt as the polite request and server rules as the backstop.

Quick Answers

  • Robots.txt tokens are matched per crawler; one token's rule does not cover another.
  • Training tokens (GPTBot, ClaudeBot, Google-Extended) and search tokens (OAI-SearchBot, PerplexityBot, Claude-SearchBot) serve different jobs.
  • Google-Extended does not affect Search ranking or snippets, per Google's documentation.
  • Content-Signal is a Cloudflare draft directive with values search, ai-train and ai-input.
  • Robots.txt is voluntary under RFC 9309; server rules enforce what the file requests.
  • Test every edit with a robots.txt tester and confirm in server logs.

Key Facts at a Glance

  • RFC 9309 (2022) defines robots.txt as a voluntary convention, not an access control.
  • GPTBot, ClaudeBot and Google-Extended are training-related tokens; OAI-SearchBot, PerplexityBot and Claude-SearchBot are search-answer tokens.
  • Google's own documentation says Google-Extended does not affect Search ranking or snippets.
  • Content-Signal is a draft proposal from Cloudflare, with values search, ai-train and ai-input.
  • Named groups win over the * group, so a wildcard rule does not override a token rule.
  • Robots.txt can be ignored; server or edge rules are the enforcement layer.
How do I block GPTBot in robots.txt?

Add a group with User-agent: GPTBot followed by Disallow: /. That stops OpenAI's training crawler from fetching your pages. It does not block OAI-SearchBot, which needs its own group.

What is the difference between GPTBot and OAI-SearchBot?

OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search. Blocking GPTBot refuses training use; blocking OAI-SearchBot removes your pages from that search crawl, so they are less likely to be cited in answers.

Does blocking Google-Extended hurt my SEO?

Google's documentation says Google-Extended does not affect Search ranking or snippets. It controls Gemini training and grounding use, not Googlebot's search crawl. Blocking it is a training decision, not a ranking one.

What is the content-signal directive?

It is a line in robots.txt, proposed by Cloudflare, that states how content may be used. The draft values are search, ai-train and ai-input. It is not an IETF standard, so treat it as a request that services may honour.

Should I allow or block AI crawlers?

It depends on your model. If you want AI answers to quote and link your pages, keep search-answer tokens open. If you sell content and refuse training use, close the training tokens. Many sites run both policies at once.

Data sources: the standards and technical parameters referenced in this article are based on the corresponding national/industry standards and the manufacturer's factory QC records; the current in-force version prevails.