Robots.txt for AI Crawlers: Token Guide

You spot your pages quoted in an AI answer and want to know which bot fetched them. Robots.txt is where you start. It's a plain-text file at your site root that lists user-agent tokens and Allow/Disallow rules. For AI crawlers, the token you pick decides whether a bot can fetch your pages for search, for training, or for both.

Here's the short answer: each AI crawler has its own user-agent token, so you write a separate group for each one. GPTBot is OpenAI's training crawler. OAI-SearchBot is OpenAI's search crawler. ClaudeBot belongs to Anthropic, PerplexityBot to Perplexity, and Google-Extended is the token that controls Google's use of your content for AI training and grounding. Blocking one doesn't block the others.

Robots.txt is a set of instructions, not a lock. A crawler that follows the Robots Exclusion Protocol will obey it. One that ignores it won't. RFC 9309 (2022) defines the protocol and says the rules are voluntary, so crawlers should honor them but nothing forces them to.

An AI crawler goes through four steps before it cites you

First, the crawler fetches robots.txt and reads the rules for its token. Second, it fetches the pages you allow. Third, the AI system indexes or stores that content. Fourth, when someone asks a question, the system pulls relevant content and may cite it.

Any step can break. If robots.txt blocks the crawler, there's no fetch. If an allowed page returns a server error, the crawler may skip it. If the content is fetched but not indexed, it can't be retrieved. If it's retrieved but not cited, the answer may lean on other sources.

This is why blocking a training crawler isn't the same as blocking a search crawler. OpenAI's documentation describes GPTBot as its training-data crawler and OAI-SearchBot as the crawler that helps surface sites in ChatGPT search. Block GPTBot and allow OAI-SearchBot, and you keep the door open for citations while limiting training use.

Treat the table as a starting point. Providers can change crawler behavior, so check their own documentation before you publish a policy.

Pick your token by deciding what you want first

Start with the goal. Want AI search citations but less training use? Allow the search crawlers and block the training crawlers. Want out of AI systems entirely? Block every known AI token.

OpenAI's documentation makes the split clear. GPTBot collects training data. OAI-SearchBot feeds search results in ChatGPT. A site that blocks GPTBot but allows OAI-SearchBot can still show up in ChatGPT search results.

Anthropic's documentation names ClaudeBot as its crawler and says it uses it to improve its models. Blocking ClaudeBot may reduce your content's presence in Claude answers, though the exact effect depends on the retrieval system.

Perplexity's documentation lists PerplexityBot as the crawler that fetches pages for its answer engine. Blocking it may reduce citations in Perplexity answers.

Google-Extended is not a search crawler. Google's documentation says it lets publishers control whether their content is used for training and grounding in Gemini, and that it does not affect Google Search ranking or inclusion.

Write one user-agent group per token, or the rules won't do what you expect

Put the token on its own line, then the Allow or Disallow rules. A group starts with a User-agent line and ends at the next User-agent line or the end of the file.

To block GPTBot but allow OAI-SearchBot, write two groups:

User-agent: GPTBot\nDisallow: /\n\nUser-agent: OAI-SearchBot\nAllow: /

That tells GPTBot to stay out and OAI-SearchBot to come in. To block all AI training crawlers but keep search crawlers, list each training token with Disallow: / and each search token with Allow: /.

A common mistake is to write User-agent: * and assume it covers every AI bot. It doesn't. The wildcard group applies only to crawlers that don't have their own group. A specific GPTBot group overrides the wildcard for GPTBot.

Another mistake is blocking a path your pages need to render. If your site relies on JavaScript to show content, blocking /js/ may stop crawlers from seeing the page. Check your robots.txt against how your site is built.

Content-Signal states a preference, but it isn't a standard

Content-Signal is a proposed set of preference signals you can add to robots.txt. Lines like Content-Signal: search=yes, ai-train=no express how content may be used. It is not part of RFC 9309, and adoption is not universal.

The proposal comes from a group of publishers and is documented in public drafts. Because it isn't a standard, crawlers may ignore it. Treat it as a statement of preference, not a technical control. Add it if you want your policy on record, but don't rely on it alone.

The syntax is still evolving, so check the current draft before you publish. If a provider doesn't support it, the line has no effect.

Some bots ignore robots.txt, so add other controls

Some crawlers don't follow robots.txt. They may be scrapers, or they may be run by parties that don't honor the protocol. RFC 9309 says crawlers should follow robots.txt, but it can't enforce compliance.

For non-compliant bots, use other controls. A Web Application Firewall (WAF) can block requests by user-agent or IP range. Rate limiting can reduce load. Authentication can keep private content out of reach. None of these is perfect, but they add layers.

Check your server logs to see which user-agents actually fetch your pages. If you see a token that isn't in your robots.txt, decide whether to add a group for it. Log analysis is the only way to know what's really happening.

Robots.txt controls crawling; meta tags and HTTP headers control indexing

Robots.txt works at the site level. Meta tags and HTTP headers work at the page level. A noindex meta tag tells search engines not to index a page, but it doesn't stop a crawler from fetching it. An X-Robots-Tag HTTP header does the same at the protocol level.

For AI crawlers, robots.txt is the main control. Some AI systems may respect noindex, but there's no public promise. Use robots.txt to block fetching, and meta tags or headers as extra signals.

CDNs and caching can override robots.txt. If a CDN serves a cached copy of robots.txt, your changes may not take effect right away. Check the cache headers and purge the cache after you update the file.

Run this checklist before you publish a policy

Use these steps to verify your robots.txt for AI crawlers.

• Fetch robots.txt from the live site and confirm it returns HTTP 200.
• Check that each AI token has its own group and that the rules are correct.
• Confirm the wildcard group doesn't accidentally block allowed paths.
• Test with a crawler simulator or a search engine's robots.txt tester.
• Check server logs after 48 hours to see which tokens are fetching pages.
• Review CDN and cache settings so the file isn't stale.

No public figure says how quickly AI crawlers re-read robots.txt. Expect variation by provider.

Key Facts at a Glance

• Robots.txt is a voluntary protocol defined in RFC 9309 (2022).
• GPTBot is OpenAI's training crawler; OAI-SearchBot is its search crawler.
• ClaudeBot is Anthropic's crawler; PerplexityBot is Perplexity's crawler.
• Google-Extended controls AI training use, not Google Search ranking.
• Content-Signal is a proposed preference signal, not a standard.
• Always verify with server logs and a robots.txt tester.

Frequently Asked Questions