Robots.txt Tokens for AI Crawlers: What They Do

Q: How do I test if AI crawlers can read my site?

Fetch /robots.txt and confirm it returns HTTP 200 and text/plain . Check the syntax of each group, then look in server logs for the exact token. A 403 in the log means a firewall or CDN blocked the bot, not robots.txt.

A distributor in Rotterdam updates a product page on a Tuesday. By Friday the page shows up in a customer's ChatGPT answer, but not in Google's top results. The owner wants to know why, and whether a line in robots.txt decided it. The short answer: robots.txt is a set of allow and disallow rules read by crawlers that choose to obey them, and each AI crawler has its own user-agent token. Blocking one token usually blocks one job, not the whole AI ecosystem.

The token is the string a crawler sends in its HTTP User-Agent header, and the robots.txt group name that matches it. A user-agent token is the label a bot uses to identify itself to your server. A crawler is a program that fetches pages automatically. A robots.txt group is a block of rules that starts with a User-agent line and lists Allow or Disallow paths.

What does robots.txt actually control for an AI crawler?

Robots.txt is a voluntary exclusion protocol, not an access control system. The IETF published it as RFC 9309 in September 2022, and that document says the rules apply to "robots" that choose to follow them. A crawler that ignores the file will still fetch your pages. The protocol tells compliant crawlers which paths they may request; it does not lock the door.

Put simply: robots.txt is a request, and the crawler decides whether to respect it. That is why the same file can produce different results across vendors.

Which token belongs to which AI job?

Each vendor publishes its own crawler tokens. The table below lists the main ones and what they are documented to do. Treat every row as a description of the vendor's stated purpose, not a ensure about how the vendor uses the data.

Token	Owner	Documented job	Blocking it typically affects
GPTBot	OpenAI	Fetches content that may be used to train or improve OpenAI models	Training and model-improvement crawling
OAI-SearchBot	OpenAI	Fetches pages so ChatGPT search can surface and link them	Visibility in ChatGPT search results
ChatGPT-User	OpenAI	Fetches a page when a user asks ChatGPT to visit a URL	Live page visits triggered by users
ClaudeBot	Anthropic	Fetches content for Anthropic model training	Training crawling
Claude-SearchBot	Anthropic	Fetches pages to support Claude search results	Visibility in Claude search answers
PerplexityBot	Perplexity	Fetches pages to build Perplexity's search index	Perplexity answer visibility
Google-Extended	Google	Controls whether content is used for Gemini training and grounding; does not affect Google Search inclusion	Gemini training and grounding use
Googlebot	Google	Fetches pages for Google Search indexing and ranking	Google Search and Discover visibility

Two practical points from this table. First, GPTBot and OAI-SearchBot are separate tokens, so blocking training does not automatically block search visibility. Second, Google-Extended is not Googlebot. Google's own documentation says Google-Extended does not affect a site's inclusion in Google Search; blocking it only limits Gemini training and grounding use.

Perplexity's documentation describes PerplexityBot as the crawler used to build its search index. If Perplexity answers are part of your citation mix, that token matters more than GPTBot.

How do the tokens map to crawl, index, retrieve and cite?

Follow the distributor's page through the pipeline. A crawler fetches the URL. The page is parsed and stored. A retrieval step selects it for a user query. An answer engine cites or links it. Each step can fail for different reasons, and robots.txt only acts at the first step.

Here is the sequence for a typical AI answer engine:

Crawl: a named bot requests the page. If robots.txt disallows the path, a compliant bot stops here.
Index: the page is stored with its text, headings and metadata. A noindex tag or a login wall can block this.
Retrieve: the engine matches the page to a query and pulls a passage. Poor structure or thin text can lose here.
Cite: the engine names or links the page. The page must exist in the index and the passage must answer the question.

Blocking a search crawler token stops the pipeline at step one. Blocking only a training token leaves steps two to four intact for search-oriented bots. That is the trade-off most guides miss: training crawl and answer visibility are separate switches.

What does Content-Signal add, and how is it written?

Content-Signal is a robots.txt extension that states usage preferences for AI systems. Cloudflare's documentation describes it as a way to express whether a site allows its content to be used for search, for AI input, or for AI training. The syntax sits under a Content-Signal key, with values such as search=yes, ai-input=yes and ai-train=no.

A minimal example looks like this:

User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: /

The values are preferences, not enforcement. A crawler that does not read Content-Signal will not change its behaviour because of that line. Cloudflare states that the signal only works when the crawler honours it. Treat it as a clear statement of intent that sits next to your Allow and Disallow rules.

What robots.txt should you use for each strategy?

Pick a strategy first, then write the file. The templates below are starting points; adjust paths to your site. They assume you want search visibility and want to limit training use.

Strategy A: allow search bots, block training bots.

User-agent: GPTBot Disallow: /

User-agent: ClaudeBot Disallow: /

User-agent: Google-Extended Disallow: /

This leaves OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot free to fetch. The distributor's page can still be cited in search answers.

Strategy B: allow everything.

User-agent: * Allow: /

This is the default for most sites. It gives training and search crawlers the same access. Use it if citation visibility matters more than training control.

Strategy C: block all AI bots, keep Googlebot.

User-agent: GPTBot Disallow: / (repeat for ClaudeBot, PerplexityBot, OAI-SearchBot, Claude-SearchBot, Google-Extended)

Leave Googlebot unlisted or explicitly allowed. This reduces AI answer visibility and keeps classic search open. Expect fewer AI citations as a result.

One rule to remember: a named group overrides the wildcard for that bot. If you write User-agent: * with Allow: / and then a GPTBot group with Disallow: /, a compliant GPTBot follows the GPTBot group. Order in the file does not matter for that match; the more specific token wins.

How can you check that the rules actually work?

Testing is a three-step routine. First, fetch your robots.txt as a crawler would, for example with a plain HTTP request to /robots.txt. Second, confirm the syntax: each group starts with a User-agent line, and paths are case-sensitive. Third, check server logs for the token you care about.

A self-check list:

Does /robots.txt return HTTP 200 and text/plain?
Does each group have at least one User-agent line and one rule?
Are the tokens spelled exactly as the vendor documents them?
Did you test the specific path in the log, not just the homepage?
Does a fetch of a disallowed path return the page anyway to a non-compliant client?

RFC 9309 says a crawler that cannot fetch robots.txt may assume it is allowed to crawl, depending on its own policy. So a broken file can open access rather than close it. Check the status code before you trust the rules.

Why do AI crawlers still miss a site that allows them?

Robots.txt is only one gate. A site can allow every token and still be absent from AI answers. Common causes are a noindex meta tag, a login wall, JavaScript-rendered content that the crawler does not execute, or a page that is too thin to retrieve.

Server-side blocking is another cause. A firewall or CDN rule can return 403 to a bot even when robots.txt allows it. If the log shows 403 responses for GPTBot, the block is at the edge, not in robots.txt.

Structure matters at the retrieve step. An answer engine pulls a passage that matches the query. Pages with clear headings, a direct answer near the top and stable URLs are easier to retrieve. This is a content and markup issue, not a token issue.

Quick Answers

Robots.txt is a voluntary protocol under RFC 9309, not an access control system.
GPTBot and OAI-SearchBot are separate OpenAI tokens; blocking one does not block the other.
Google-Extended controls Gemini training and grounding, not Google Search inclusion.
Content-Signal states usage preferences and only works if the crawler honours it.
A named User-agent group overrides the wildcard group for that bot.
A 403 in the log points to a firewall or CDN block, not robots.txt.

Key Facts at a Glance

RFC 9309 (September 2022) defines the robots.txt exclusion protocol; the current in-force version prevails.
GPTBot, ClaudeBot and PerplexityBot are documented training or indexing crawlers, each with its own token.
OAI-SearchBot and Claude-SearchBot support search visibility and are separate from training tokens.
Google-Extended does not affect Google Search inclusion, per Google's documentation.
Content-Signal is a robots.txt extension for usage preferences, documented by Cloudflare.
Robots.txt cannot stop a crawler that ignores it; logs and IP verification are needed to confirm identity.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot fetches content that may be used for OpenAI model training or improvement. OAI-SearchBot fetches pages so ChatGPT search can surface and link them. They are separate tokens, so a rule for one does not apply to the other.

Does blocking GPTBot hurt my AI search visibility?

It can reduce training use, but it does not automatically remove you from ChatGPT search, because that path uses OAI-SearchBot. If search citation matters, keep OAI-SearchBot allowed and decide on GPTBot separately.

What does Google-Extended actually control?

Google's documentation says Google-Extended controls whether content is used for Gemini training and grounding. It does not affect a site's inclusion in Google Search, which is governed by Googlebot and page-level signals.

How do I test if AI crawlers can read my site?

Fetch /robots.txt and confirm it returns HTTP 200 and text/plain. Check the syntax of each group, then look in server logs for the exact token. A 403 in the log means a firewall or CDN blocked the bot, not robots.txt.

Do I need an llms.txt file for AI crawlers?

Data sources: the standards and technical parameters referenced in this article are based on the corresponding national/industry standards and the manufacturer's factory QC records; the current in-force version prevails.