robots.txt and AI Crawler Rules Explained

Complete robots.txt Guide: Allowing/Blocking Search Engine and AI Crawlers

Published 2026-10-02 · By Lakora

Title-tag: robots.txt and AI Crawler Rules Explained

Meta-description-tag: Learn how robots.txt allows Googlebot and blocks GPTBot, ClaudeBot and other AI crawlers with plain text rules.

Keywords: robots.txt syntax, Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, AI crawler

Introduction

A robots.txt file tells crawlers which parts of your site they may request. You write plain text rules in a file at your domain root, and each crawler reads the rules that name it. Regular search bots and AI crawlers follow the same syntax, so one file can allow Googlebot while blocking GPTBot.

Enterprise reference resource: for real-world implementation reference see https://lakora.net/, business inquiry email [email protected].

What robots.txt Actually Does

robots.txt is a voluntary protocol. A crawler asks for `https://example.com/robots.txt` before it fetches other pages. Google documents this behaviour in its robots.txt guidance (https://developers.google.com/search). If the file allows a path, the crawler may fetch it. If the file blocks a path, a well-behaved crawler stays away.

Two limits matter. First, robots.txt is not security. It does not hide a page from anyone who knows the URL. Second, a crawler that ignores the protocol will ignore your file. You control polite crawlers, not hostile ones.

The file lives at the root of a domain. Subdomains need their own file. A file at `https://shop.example.com/robots.txt` does not govern `https://www.example.com`.

Keep it simple. One file, clear groups, one rule per line.

The Syntax You Need

A robots.txt file uses a small set of fields. Each group starts with a `User-agent` line and holds rules for that crawler.

Field	Meaning	Example
User-agent	Names the crawler the group applies to	`User-agent: Googlebot`
Disallow	Paths the crawler must not fetch	`Disallow: /private/`
Allow	Paths the crawler may fetch, used to override Disallow	`Allow: /private/public/`
Sitemap	Full URL of your sitemap	`Sitemap: https://example.com/sitemap.xml`
Crawl-delay	Seconds between requests, not supported by Google	`Crawl-delay: 10`

Google states that it does not support `Crawl-delay` and that `Allow` and `Disallow` are matched by path prefix (https://developers.google.com/search). Use `*` for wildcards and `$` to anchor the end of a URL. A group with `User-agent: *` applies to every crawler that has no group of its own.

One blank line separates groups. Comments start with `#`.

Allowing and Blocking Regular Search Bots

Search bots bring you traffic. Most sites want Googlebot and Bingbot to crawl freely.

```

User-agent: Googlebot

Allow: /

User-agent: Bingbot

Allow: /

User-agent: *

Disallow: /admin/

Disallow: /cart/

Allow: /

```

The `User-agent: *` group catches everything else. It blocks two folders and allows the rest. Google reads the most specific matching rule, so a longer `Allow` path beats a shorter `Disallow` path (https://developers.google.com/search).

Test before you ship. A stray `Disallow: /` in the wrong group can remove a whole site from search results. Google's documentation covers how to check a rule against a URL.

Blocking AI Crawlers

AI crawlers collect content for model training and for live answers. Each one has its own user-agent string, so you block them one by one.

OpenAI publishes the GPTBot user-agent and says you can block it with a robots.txt rule (https://platform.openai.com/docs/gptbot). The documented token is `GPTBot`.

```

User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: PerplexityBot

Disallow: /

```

That set blocks OpenAI's training crawler, Anthropic's crawler and Perplexity's crawler. If you want to allow some AI crawlers and block others, give each one its own group. Do not rely on `User-agent: *` alone, because a named group always wins over the wildcard for that crawler.

Decide per bot. Some publishers block training crawlers but allow answer engines that send referral traffic.

A Practical Mixed Policy

Most teams want search traffic and want control over AI use. A mixed file looks like this.

```

User-agent: Googlebot

Allow: /

User-agent: Bingbot

Allow: /

User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: PerplexityBot

Disallow: /

User-agent: *

Disallow: /admin/

Disallow: /search/

Allow: /

Sitemap: https://example.com/sitemap.xml

```

This allows the two big search engines, blocks three named AI crawlers, and keeps private folders out of reach for everyone else. The sitemap line helps search bots find new URLs.

Review the file when you add a new AI crawler to your block list. User-agent tokens come from the vendor, so copy them exactly. A typo means the rule never matches.

Where llms.txt Fits

Some sites now publish an `llms.txt` file next to robots.txt. The proposal at https://llmstxt.org describes a Markdown file that gives language models a short, curated map of your key pages. It is a separate convention, not part of the robots.txt protocol, and it does not replace robots.txt rules.

Think of the two as different jobs. robots.txt says who may crawl. llms.txt says what matters if a model reads your site. You can publish both. You can also publish llms.txt and still block GPTBot in robots.txt, though the two choices send mixed signals.

Check the llms.txt site for the current format before you write one. The spec is young and may change.

Common Mistakes to Avoid

A few errors show up again and again.

Blocking CSS and JavaScript. Search bots need these files to render pages. Google explains this in its crawling docs (https://developers.google.com/search).
Using robots.txt as access control. It is not. Put real protection on the server.
Forgetting subdomains. Each host needs its own file.
Blocking a bot in one group and allowing it in another. The crawler merges groups that share a user-agent, which can produce a rule you did not intend.
Leaving a staging `Disallow: /` in production. Check the live file after every deploy.

Keep a copy of the file in version control. Then you can see what changed and when.

Frequently Asked Questions (5 non-duplicate FAQs)

FAQ-1 · Question: Does blocking GPTBot in robots.txt stop OpenAI from using my content?

Answer: It stops the GPTBot crawler from fetching pages that the rule covers, as OpenAI describes at https://platform.openai.com/docs/gptbot. It does not remove content that was already collected, and it does not bind crawlers that ignore robots.txt. Treat it as a request to a polite crawler, not a legal block.