Robots.txt Tokens for AI Crawlers: A Practical Walkthrough

A content lead at a small B2B software company opens Google Search Console and sees a warning: some pages are blocked by robots.txt. The site's robots.txt was written in 2019, before GPTBot and ClaudeBot existed. The team wants ChatGPT and Perplexity to cite them, but they also don't want their support docs used to train models. They need to know what each token does before they change a line.

Robots.txt is a plain-text file at the root of a domain that tells automated crawlers which paths they may request. It is voluntary: a crawler has to choose to follow it. The tokens below are user-agent names that AI companies publish so site owners can give per-bot instructions.

Robots.txt controls crawling, not indexing or citation

A crawler that respects the file will not fetch a disallowed URL. If the URL is already known from links or sitemaps, it can still appear in an index without its content being read. That gap matters for AI answers, because a model can cite a page only if it can read the page.

The file is a set of groups. Each group starts with a User-agent line, followed by Allow or Disallow rules. A crawler picks the group whose user-agent matches it, or the * group if no specific match exists. Rules are matched as path prefixes unless they contain wildcards. The original specification is RFC 9309 (2022), which made the 1994 convention an internet standard.

Put simply: robots.txt is a request, not a lock. A bot that ignores it can still fetch your pages at the network level. That is why server-side blocking or a web application firewall is the harder control.

Each AI company uses separate tokens for separate jobs

One bot may collect pages for model training; another may fetch pages to answer a search query. Blocking one does not automatically block the other. The table below lists the main tokens and what each is documented to do.

The operator documentation is the authority here: OpenAI publishes its crawler list on platform.openai.com, Anthropic on its support site, Perplexity on its documentation, and Google on developers.google.com/search. Token lists change, so check the operator page before you ship a rule.

Write one block per bot to keep the blast radius small

A per-bot block is safer than one generic block because it keeps the blast radius small. Each block starts with the token, then the paths. The examples below are syntax patterns, not recommendations for your site.

To block GPTBot from training on your blog but leave the rest of the site open:

User-agent: GPTBot\nDisallow: /blog/\nAllow: /

To block ClaudeBot from the whole site while leaving other crawlers alone:

User-agent: ClaudeBot\nDisallow: /

To let PerplexityBot read public pages but keep it out of account areas:

User-agent: PerplexityBot\nDisallow: /account/\nAllow: /

To stop Google-Extended from using content for Gemini training without touching Googlebot:

User-agent: Google-Extended\nDisallow: /

Order matters. Most crawlers evaluate the longest matching path, and Allow beats Disallow when the match length is equal. RFC 9309 describes the matching rules; Google's own documentation explains how Googlebot applies them. Test any change with the operator's tester or a generic robots.txt validator before publishing.

Content-Signal records preferences but does not enforce them

Content-Signal is a set of preferences added to robots.txt by Cloudflare in September 2025, expressed as a comment-style line. It is not part of RFC 9309 and it does not change crawl behaviour on its own. It records what you prefer your content be used for, so participating services can read it.

The three signals are search (allow indexing for search), ai-input (allow use as input to AI answers) and ai-train (allow use for model training). Each takes yes or no. A typical line looks like this:

Content-Signal: search=yes, ai-input=yes, ai-train=no

Treat Content-Signal as a declaration, not an enforcement. Cloudflare's own announcement says it works with participating crawlers; a bot that does not read it will not obey it. Pair it with per-bot Disallow rules and, where needed, server-level blocking.

Four common mistakes break the file

The most common mistake is a Disallow: / under User-agent: * that was meant to block one bot. That rule blocks every crawler that has no specific block, including search engines. The second mistake is blocking Google-Extended and assuming Google Search will stop indexing; it will not, because Googlebot is a separate token.

The third mistake is blocking GPTBot and expecting ChatGPT search citations to disappear. OAI-SearchBot is the token for search visibility, so the two jobs need separate decisions. The fourth is a missing newline between groups, which merges two blocks into one and applies the wrong rules.

• Check that each User-agent line starts a new block.
• Check that every Disallow path starts with a slash.
• Check that no rule accidentally covers the whole site.
• Check the live file, not a draft, because CDNs can cache it.

Verify syntax first, then behaviour in the server log

Verification has two parts: syntax and behaviour. For syntax, paste the file into a robots.txt validator or the operator's tester. For behaviour, check the server log for requests from the bot's user-agent and confirm the status codes match your intent. A blocked path should return 403 or 404 to that bot; an allowed path should return 200.

If the site is new, expect crawl activity to grow over the first weeks rather than appear on day one. To verify a change, compare the crawl log before and after the edit, filtered by user-agent. RFC 9309 says crawlers should check robots.txt before fetching, but the cache period is implementation-defined, so there is no public figure for how quickly each operator re-reads it.

Use a four-step checklist and keep the file in version control

A workable checklist has four steps. First, list which bots you want to allow for search and which you want to exclude from training. Second, write one block per token, with the narrowest paths that do the job. Third, add a Content-Signal line that matches those choices. Fourth, test the live file and check the log after 48 hours.

Keep a copy of the file in version control so you can see when a rule changed. If a page must never be fetched, use authentication or a server rule rather than relying on robots.txt. The file is a signal; the server is the control.

Quick Answers

• Robots.txt is voluntary; it controls crawling requests, not indexing or citation.
• GPTBot and ClaudeBot are documented as training crawlers; OAI-SearchBot and PerplexityBot serve search and answers.
• Google-Extended controls Gemini training and grounding, not Google Search indexing.
• Content-Signal is a preference line added by Cloudflare in September 2025; it is not part of RFC 9309.
• Test the live file and check server logs by user-agent before and after any change.

Key Facts at a Glance

• RFC 9309 (2022) is the current standard for robots.txt; the in-force version prevails.
• Each AI operator publishes its own token list; check the operator page before shipping a rule.
• Blocking a training bot does not block a search bot, and the reverse is also true.
• Content-Signal uses three keys: search, ai-input, ai-train.
• Server-level blocking is the harder control when robots.txt is not enough.
• Typical parameters and behaviour are implementation-defined; test on your own site.

Data sources: the standards and technical parameters referenced in this article are based on the corresponding national/industry standards and the manufacturer's factory QC records; the current in-force version prevails.

Sources (checked 2026-10-27): RFC 9309: Robots Exclusion Protocol (IETF) · Google Search Central: How to write a robots.txt file.