Robots.txt, Content-Signal, llms.txt: Make Them Agree

A publisher's site allowed GPTBot in robots.txt, sent a Content-Signal header saying ai-train=no, and ran an llms.txt that invited AI systems to use the content. Three files, two messages. That conflict is common. Here's the short version: robots.txt is the access control file a crawler reads first. Content-Signal is a declared preference about AI use. llms.txt is a plain-text map for language models. This article gives you an inspection order to make them agree.

A real three-way conflict is three different answers to one question

Take a site that publishes research summaries. Its robots.txt has User-agent: GPTBot followed by Allow: /, so the crawler is allowed in. The same site sends a Content-Signal response header that sets ai-train=no, so the site says its content must not be used for training. The llms.txt file opens with a friendly note: "AI systems may use these pages to answer questions." Three files, three answers.

Nothing crashes. You just get ambiguity. A crawler that follows robots.txt will fetch the page. A system that reads Content-Signal may record a training restriction. A model that reads llms.txt may treat the content as available. No mechanism enforces agreement between the three. The site owner has to do that.

This is a policy consistency problem, not a syntax problem. Each file can be valid on its own and still contradict the others. The checklist below walks through the files in the order a crawler meets them.

robots.txt comes first, and a block there makes the other two files irrelevant

A crawler typically requests /robots.txt before fetching content, as RFC 9309 (the Robots Exclusion Protocol) describes. If robots.txt blocks the path, the crawler stops. Content-Signal and llms.txt never get read.

Content-Signal is a set of preferences carried in HTTP response headers, proposed by Cloudflare in 2025. It travels with the page response, not before it. llms.txt is a proposed markdown file at /llms.txt that lists key pages for language models. No IETF standard covers llms.txt.

So the inspection order is robots.txt first, then the response headers on a sample page, then llms.txt. If step one blocks the crawler, steps two and three are decoration.

Audit robots.txt by checking tokens, groups and collisions

Start with the user-agent groups. Each group is a set of rules for one named crawler, and the group name must match the token exactly. OpenAI documents GPTBot for training crawls and OAI-SearchBot for search. Anthropic documents ClaudeBot and Claude-SearchBot. Check each token against the provider's own documentation, not a third-party list.

Then look for rule collisions. A wildcard User-agent: * group applies to any crawler without its own group. When a named group exists, parsers use the named group and ignore the wildcard for that crawler. A specific group overrides the general one. That's where silent conflicts start.

Check the classic errors too: a Disallow: / under the wrong group, a missing colon, or a path that doesn't start with a slash. RFC 9309 defines the syntax, and a malformed line may be ignored. Test the live file in Google Search Console's robots.txt report and in Bing Webmaster Tools. Both show the parsed result, not just the raw text.

Access and permission are different questions, and that's where the files drift apart

Content-Signal uses three fields: ai-train, ai-input and search. Each can be set to yes or no. The conflict shows up when robots.txt allows a crawler but Content-Signal says ai-train=no. Robots.txt answers "may you fetch this?" Content-Signal answers "may you use it for this purpose?" A site can say yes to the first and no to the second.

llms.txt adds a third layer. It isn't a permission file. It's a curated list. If robots.txt blocks /research/ but llms.txt links to a page under /research/, the link is unusable. The crawler can't fetch the target.

Fix the direction of the conflict first. Decide the policy in one place, then write the other two files to match it. Start with the most restrictive file and relax it, rather than the reverse. That way you never publish a permissive file while a restrictive one is still live.

Pick one of three policies and apply it everywhere

Most sites land on one of three positions. Choose one and apply it to all three files.

• Open: robots.txt allows AI crawlers, Content-Signal sets ai-train=yes, llms.txt lists key pages. Use this when you want citations and training use.
• Search only: robots.txt allows search crawlers and blocks training crawlers, Content-Signal sets ai-train=no and search=yes, llms.txt lists pages for retrieval. Use this when you want answers but not training.
• Closed: robots.txt blocks AI crawlers, Content-Signal sets all fields to no, llms.txt is absent or minimal. Use this when you want no AI use at all.

The middle option is the one most sites actually want, and it's the one most often broken. A site blocks GPTBot in robots.txt but leaves OAI-SearchBot allowed. That's a search-only policy. If Content-Signal still says ai-train=yes, the files disagree.

Write the policy down in one sentence before you edit any file. For example: "Search and retrieval allowed; training not allowed." Then check each file against that sentence.

Run the consistency check in six steps

Here's the order to use on a live site. A small site takes about an hour.

1. Fetch /robots.txt and list every user-agent group. Note which AI tokens are allowed and which are blocked.
2. Fetch a sample page and read the response headers. Record the Content-Signal values.
3. Fetch /llms.txt if it exists. List the URLs it points to.
4. Cross-check: every URL in llms.txt must be fetchable under robots.txt. Every allowed crawler in robots.txt must match the Content-Signal intent.
5. Test with the platform tools: Google Search Console robots.txt report, Bing Webmaster Tools, and a header inspector.
6. Log the result with a date. Re-check after any policy change.

Step four is the one most sites skip. A link in llms.txt to a blocked path is a dead entry. The file looks fine and does nothing.

Test with the platform tools, then spot-check a few pages

Use the tools each platform provides. Google Search Console has a robots.txt report that shows the parsed rules. Bing Webmaster Tools has a similar tester. For headers, use the browser developer tools or a header-checking service. No official validator exists for llms.txt, so check the file by hand against robots.txt.

A practical test: pick one page you care about. Confirm it's allowed in robots.txt. Confirm the Content-Signal header on that page matches your stated policy. Confirm the page appears in llms.txt only if it's fetchable. If all three pass, the policy is consistent for that page. Repeat for two or three representative pages.

No public figure exists for how many sites have this conflict. What's clear is that different people maintain these files on many sites, so drift is likely.

Key Facts at a Glance

• robots.txt is the access control file; Content-Signal is a declared preference; llms.txt is a map. They answer different questions.
• The inspection order is robots.txt, then response headers, then llms.txt, because a blocked crawler never reads the other two.
• A specific user-agent group overrides the wildcard group for that crawler, per RFC 9309 parsing behaviour.
• Every URL in llms.txt must be fetchable under robots.txt, or the entry is dead.
• Write the policy in one sentence first, then make all three files match it.