Robots.txt, Content-Signal and llms.txt: Make Them Agree

A publisher blocks GPTBot in robots.txt, adds Content-Signal to allow search but not training, and publishes an llms.txt that invites AI assistants to read the whole site. Three files, three different answers. That's what a policy conflict looks like.

Don't pick one file and trust it. Make every place that states crawler rules say the same thing, then check with real fetches. A crawler policy is consistent when robots.txt, HTTP headers, HTML meta tags, Content-Signal and llms.txt all give one answer for one user-agent.

Each file controls a different thing, and that's where conflicts start

Robots.txt is a plain-text file at the root of a host. It tells crawlers which paths they may fetch. The Robots Exclusion Protocol is described in RFC 9309 (September 2022). That protocol only covers fetching. It says nothing about what happens to content after it's fetched.

Content-Signal is a set of preferences written inside robots.txt, using comment-style lines such as Content-Signal: search=yes, ai-train=no. Cloudflare documents this format in its Content Signals Policy. The signals describe how content may be used, not whether it can be fetched. They're a stated preference, not an access control.

Llms.txt is a proposed convention: a Markdown file at /llms.txt that gives AI systems a short map of a site's key pages. It's not a standard. No IETF RFC, W3C recommendation or vendor documentation makes it binding, so treat it as a hint that some tools read and others ignore.

HTTP headers and HTML meta tags are the third layer. X-Robots-Tag in a response header and <meta name="robots"> in the HTML both carry directives per URL. Google documents both in its crawler and indexing docs, and states that noindex is honoured only when the crawler can fetch and parse the page.

Most conflicts are semantic mismatches, not syntax errors

One file talks about fetching, another about usage, a third about discovery. A CDN rule can quietly block the request before any of them are read.

Here's the pattern to look for first. Robots.txt allows /articles/ for all agents. The page HTML carries noindex. Content-Signal says ai-train=no. Llms.txt lists that same article as a recommended source. Every statement is valid on its own, and together they contradict each other.

There's a second, less obvious layer. Many sites sit behind a CDN or WAF that applies a bot score or a managed rule set. If that layer returns 403 to GPTBot, the crawler never reads robots.txt at all. Your carefully written rules don't matter, because the request stopped at the edge.

When two layers disagree, the more restrictive one usually wins in practice, but not always for the reason people expect. A Disallow in robots.txt stops the fetch, so the crawler never sees the meta tag. A noindex tag only works if the page is fetchable. Blocking and de-indexing are different operations, and mixing them up causes most of these conflicts.

Audit from the outside in, starting with the request itself

Start where the crawler starts: the request. Fetch the URL with a named user-agent and look at the status code before you read any policy text.

• Request the page as GPTBot and as a normal browser. Compare status codes. A 403 for one and 200 for the other means the edge is deciding, not your files.
• Fetch /robots.txt and confirm it returns 200 with text/plain. A 404 means everything is allowed under RFC 9309.
• Check for Content-Signal lines inside robots.txt and note which groups they sit in.
• Fetch /llms.txt and list the URLs it recommends.
• Pull the response headers for each recommended URL and look for X-Robots-Tag.
• Fetch the HTML and read the meta robots tag.
• Cross-check the lists. Any URL that's recommended in one file and restricted in another is a conflict.

This audit takes an afternoon on a site with a few hundred URLs. Scale it with a script that loops over your sitemap and records four values per URL: header directive, meta directive, robots.txt verdict, and llms.txt presence.

For the fetch step, use the user-agent strings published by each vendor. OpenAI documents GPTBot in its own crawler documentation, and Anthropic documents ClaudeBot in its support pages. Copy the exact token. A typo in a user-agent string fails silently, because the group simply doesn't match.

The earliest blocking layer wins for access; the most restrictive layer wins for indexing

Those are two different questions, and a policy that answers only one of them is half-built.

For access, the order is roughly: network and CDN rules, then robots.txt, then the HTTP response. If the CDN returns 403, nothing downstream matters. If robots.txt disallows the path, the crawler stops. If the response returns 200, the page is fetched and the header and meta directives are read.

For indexing, a noindex in either the header or the meta tag is enough to keep a page out of the index, provided the crawler can fetch it. Google's documentation states that a page blocked by robots.txt cannot be read, so its noindex cannot be seen. That's why blocking and noindexing the same URL produces the opposite of what many people intend.

Content-Signal doesn't override any of this. It records a preference. If you set ai-train=no but leave the path open, you've stated a wish, not enforced a rule. Enforcement needs the fetch blocked at a layer that actually blocks.

Allow or block AI crawlers? Decide by business model, not by default

There's no single right answer, and anyone who gives you one is skipping the trade-off. The decision depends on your business model, your content type and what you want out of AI surfaces.

Ask three questions. Does your revenue come from people visiting the page, or from being cited as a source? Does your content lose value when it's summarised without a click? Do you have a licensing or attribution arrangement that depends on a specific crawler being allowed?

If citations matter more than sessions, allowing retrieval crawlers and blocking training crawlers is a coherent position. If the content is paid or licensed, blocking both and negotiating access is coherent too. What's not coherent is publishing llms.txt that invites reading while robots.txt blocks the same agents.

Separate the crawler types in your thinking. A search crawler fetches pages to build an index that sends visitors. A training crawler collects content for model training. A retrieval crawler fetches pages at answer time to ground a response. The same user-agent group in robots.txt can treat them differently, but only if the vendor publishes distinct tokens you can name.

A consistent policy is one table, translated into files, then tested

Write the decision in a table first, then translate it into files, then test it. The table is the policy. The files are just delivery.

Take a worked example. Say you want search engines to index everything, AI answer engines to read and cite your guides, and training crawlers to stay off your paid reports. The table has three rows: search, retrieval, training. Each row gets one verdict per path group.

Now translate. Robots.txt gets one group per user-agent token, with the paths that match the table. Content-Signal lines go in the groups where a usage preference adds information. Llms.txt lists only the paths marked as listed, and never a path that appears in a Disallow line.

Then check the edges. Confirm the WAF doesn't return 403 to any agent you allow. Confirm no template injects a stray noindex. Confirm the header and meta tag agree on every URL in the llms.txt list.

Policies drift because three people can change three layers

A marketing template change can add a meta tag. A security review can tighten the WAF. A content update can add a URL to llms.txt. Nobody reviews these changes against each other.

The practical fix is a small, boring check that runs on every deploy. Fetch the top twenty URLs, record the four values, and fail the build if a URL appears in llms.txt while also carrying a restrictive directive. That single rule catches most real conflicts.

Review the policy on a fixed date. Vendors add crawler tokens and change how they treat signals, so a policy written in one quarter can be out of date the next. Put the review in the calendar with an owner's name next to it, and note which document each decision came from.

Record the sources in the policy file itself. A short comment listing the RFC number, the vendor doc URLs and the date checked makes the next review much faster. It also stops someone re-litigating a decision that was already made.

Key Facts at a Glance

• Robots.txt is governed by RFC 9309 (September 2022).
• Content-Signal is a stated usage preference inside robots.txt, documented by Cloudflare, not an access control.
• Llms.txt is a proposed convention with no formal standard behind it.
• A robots.txt Disallow stops the fetch, so meta noindex tags on that URL are never read.
• CDN and WAF bot rules act before robots.txt and can silently override the whole policy.
• Consistency means one policy table, translated into files, then verified with real fetches per user-agent.