Robots.txt AI Access Check

Paste a robots.txt and see which search engines, AI retrieval crawlers and AI training crawlers it allows or blocks. Parsed in your browser against the RFC 9309 group rules. Nothing is uploaded.

Most sites we look at do not decide this on purpose: they add a sitemap line, never list the AI crawlers, and inherit whatever the default group says. Which means either every AI crawler is welcome, or a well-meant User-agent: * / Disallow: / quietly switched off AI access entirely.

What this checks, and how

• Grouping rules (RFC 9309). Consecutive User-agent lines share one rule group, and the longest matching user agent wins. A naive line-by-line parser gets this wrong — which is why a Disallow can appear to apply to a crawler it never covered.
• Who is who. We classify each user agent by its documented purpose: search indexing (Googlebot, Bingbot), AI retrieval (OAI-SearchBot, PerplexityBot, Claude-User, ChatGPT-User), AI training (GPTBot, Google-Extended, CCBot, ClaudeBot, Applebot-Extended) and SEO data tools (AhrefsBot, SemrushBot). Blocking a training crawler does not stop its search counterpart, and the reverse is also true.
• What it can prove. This tool reads your declared policy. It cannot tell you what a crawler will actually do, whether you are indexed, or whether an AI system cited you. For what your own server saw, use the AI Crawler Log Analyzer instead.

The line worth remembering

An empty robots.txt is not neutral. It grants every crawler listed here full access by default — including the ones whose purpose is training a model, not sending you a visitor.

Frequently asked questions

Does blocking GPTBot or CCBot affect Google Search?

No. Google-Extended and GPTBot are separate user agents from Googlebot. Blocking them removes content from those training pipelines without changing search indexing or ranking eligibility.

Does a training block hurt my visibility in AI answers?

It depends which crawler you block. Training crawlers build future model weights; retrieval crawlers (OAI-SearchBot, PerplexityBot, Claude-User) fetch pages at answer time. Blocking retrieval is what removes your site from live answers; blocking training does not.

Why does my sitemap line matter here?

It does not change access; it only helps discovery. A robots.txt can be perfectly permissive and still leave a site uncrawled if nothing links to it.

Is my robots.txt sent anywhere?

No. The file is parsed in your browser with vanilla JavaScript. There is no upload, no account, and no analytics on the contents. The page itself is static.

If you would rather not maintain this by hand

We publish the policy we actually run on this site, including the groups above, in our own reports and in the live crawler view. If you want the same setup reviewed against your own logs, ask us a question.