AI Policy File Builder

Build a robots.txt and an llms.txt that say clearly which AI crawlers may read your site. Everything is assembled in your browser. No account, and nothing is uploaded.

Nothing is sent to a server.

How to deploy these files

  • Save the robots.txt output as robots.txt in your site root, so it answers at https://your-domain/robots.txt.
  • Save the llms.txt output as llms.txt in the same place.
  • Reload both URLs in a browser to confirm they are served as plain text, then re-run the AI Crawler Radar to check the result.

What this tool cannot do

  • It cannot make a crawler obey the file. robots.txt is a request that well-behaved crawlers follow, not a lock.
  • It cannot see your current robots.txt or check what your server actually serves.
  • It does not know which crawlers matter for your market, so the choices above are yours.

Questions

Should I allow or block AI training crawlers?

That is a business decision, not a technical one. Allowing training can help a model know your brand; blocking it keeps your text out of training sets. Both choices are legitimate, and this tool writes whichever you pick.

Does blocking a crawler stop it from reading my site?

Only if the crawler respects robots.txt. Reputable search and AI crawlers do. A scraper that ignores robots.txt also ignores this file, which is why server-side blocking exists.

Are retrieval agents and training crawlers different?

Yes. Retrieval agents fetch a page at the moment a user asks something and can lead to a citation. Training crawlers collect text for model training. Blocking the first group reduces your chance of being cited; blocking the second group does not.

Is this file a ranking factor or a guarantee?

No. It records your preferences. It does not guarantee indexing, citations or rich results.

Built and maintained by the same team that publishes this site. It generates text files. It does not promise rankings, rich results or citations.