A single comment line at the top of robots.txt can state, in machine-readable form, what you want to happen with your content: search indexing, AI answers, AI training. It is one of the cheapest useful things a site owner can publish, and one of the easiest to get wrong.
A Content-Signal line states a policy; robots rules and server responses are what enforce it.
Where the line lives
The line sits in robots.txt, the same file crawlers already read. On this site it reads Content-Signal: search=yes, ai-input=yes, ai-train=no, and the file then spells out which crawler groups each verdict applies to.
What it does
- It states a preference in one place, in a form a machine can read without guessing your intent from prose.
- It lets you separate retrieval from training: being read in order to answer a question is not the same as being absorbed into a model.
- It gives you something concrete to point at when someone asks what your policy is.
What it cannot do
- It cannot enforce anything. It is a declaration, and enforcement depends on the crawler choosing to respect it, plus your own edge rules for the abusive traffic that does not.
- It cannot make you visible. Allowing AI retrieval does not cause an assistant to cite you; that still depends on your content.
- It cannot fix contradictory files. A signal that disagrees with your own llms.txt is worse than no signal, because both a human and a machine now have two answers.
Keep robots.txt and llms.txt in step
This is not hypothetical. When we audited our own files, robots.txt said ai-train=no with training crawlers disallowed, while llms.txt still declared ai-train=yes and described the same crawlers as allowed. Both files were internally tidy; together they contradicted each other.
| File | Before | After |
|---|---|---|
| robots.txt | ai-train=no, training crawlers disallowed | unchanged |
| llms.txt, Content signals | ai-train=yes | ai-train=no |
| llms.txt, crawler list | training group described as allowed | not permitted, per robots.txt |
| llms.txt, groups | 4 allowed agents listed as training | moved to the retrieval group |
The fix was not a policy change. It was making the mirror actually mirror: llms.txt now says which file wins if they ever disagree, and we check the two files against each other instead of trusting that they match. If you want the same check in your browser, Robots.txt AI Access Check reads your file and reports which crawler groups it allows.
Source: worked example from this site's own files on 26 September 2026; the current versions are always at /robots.txt and /llms.txt.Does a Content-Signal line replace User-agent groups?
No. The signal states intent; the User-agent groups carry the actual Allow and Disallow rules. Publish both, and keep them consistent.
Should I block AI training crawlers?
That is a business decision, not a default. What matters is that your files say one thing, and that the same verdict appears in every machine-readable file you publish.
How do I test it?
Read robots.txt and llms.txt side by side and compare the three signals and each crawler group. If the two disagree, you are about to publish two answers to the same question.
Related reading: Five Layers of Scanner Defence, Tested on Our Own Site · Layer 5: Closing the Loop in Public
More on this site: live signals · dated reports · how to verify us