A server log is a plain text record of every request your web server answers, one line per request. Reading it for crawlers means filtering those lines by user agent, then verifying each hit against the provider's published IP ranges. Two approaches do this: manual log reading, and a log analyzer tool. Manual reading gives you proof on a small sample; a tool gives you coverage across millions of lines. Most teams need both, in that order.
1. A log hit is a claim, not a citation
The question teams actually ask is "are AI engines reading my site?" The log answers a narrower question: "did a request arrive that claims to be an AI crawler?" That is a claim, not a fact, until you verify it. A log hit is not proof of an AI citation, an AI answer, or even that a model saw your page.
Bot traffic is easy to fake. Any client can send a user agent string that says GPTBot or ClaudeBot. The user agent field is self-declared. Treat it as a starting point for filtering, never as evidence on its own.
So the job splits in two: find candidate lines fast, then prove identity on a sample. Manual reading is good at the second job. A log analyzer is good at the first.
Most web servers write logs in the Common Log Format or the Combined Log Format, both documented in the Apache HTTP Server documentation. A Combined Log Format line has seven fields: client IP, identity, authenticated user, timestamp, request line, status code, and bytes sent, plus referrer and user agent.
The fields you care about for crawler work are the client IP, the timestamp, the request line (method plus path), the status code, and the user agent. The referrer is usually empty for crawlers, which is itself a weak signal.
One trap: if your site sits behind a CDN or reverse proxy, the client IP in the log may be the proxy's IP, not the crawler's. Check your server or CDN documentation for the header that carries the real client address, often X-Forwarded-For or CF-Connecting-IP. Get this wrong and every verification step after it fails.
2. Two approaches side by side
Here is the core comparison. Neither column is "strong"; each answers a different question.
| Dimension | Manual log reading | Log analyzer tool |
|---|---|---|
| What it answers | Is this specific hit real? | How many hits, from where, over time? |
| Sample size | Small, chosen by you | Full file or a configured sample |
| Verification depth | Deep: reverse DNS, IP range check, robots.txt fetch | Shallow unless the tool supports range lookup |
| Setup cost | Low: shell tools you already have | Medium: install, parse config, storage |
| Good for | Spoofing checks, incident review | Trends, dashboards, weekly reporting |
| Main risk | You miss patterns outside your sample | You trust the user agent field as-is |
Start manual for one week, so you learn what normal looks like. Then move the counting work to a tool and keep the manual check for any hit that matters.
3. Finding candidates and proving identity
Filter by user agent first. Provider documentation lists the exact strings. OpenAI's documentation names GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic's documentation names ClaudeBot, Claude-User and Claude-SearchBot. Google's crawler documentation covers Googlebot and Google-Extended.
User agents change. Before you build a filter, open the provider's current documentation page and copy the strings from there. A filter built on a remembered string is a filter that quietly stops matching.
A simple grep on the log file is enough to start. Count lines per user agent, then look at which paths they requested. If an "AI crawler" is requesting /wp-login.php or /.env, that is not a reader. That is a scanner wearing a costume.
Verification means confirming that the IP address behind a request belongs to the provider named in the user agent. There are two accepted methods, and they differ by provider.
OpenAI publishes its crawler IP ranges in a public JSON file, and its documentation describes checking a request's source IP against that list. Anthropic publishes its IP ranges in a similar public list. Google's documentation describes forward-confirmed reverse DNS: look up the hostname for the IP, then resolve that hostname back to an IP and confirm it matches.
- Step 1: take the client IP from the log line.
- Step 2: look it up in the provider's published range list, or run the reverse DNS check where the provider uses that method.
- Step 3: record the result next to the log line: verified, unverified, or spoofed.
- Step 4: repeat on a random sample, not just the first hit you see.
Sample size matters. A sample of a few hundred hits tells you whether spoofing is present. It does not tell you the spoofing rate for the whole month. Report it as a sample finding, not a census.
4. Spoofed bots, status codes and robots.txt
A spoofed bot is a request that claims a crawler identity but fails verification. In practice, this is common enough that you should look for it on purpose. Unverified hits that name a major crawler are the first thing to check after filtering.
Spoofed traffic distorts your numbers in both directions. It inflates "AI crawler hits" if you count user agents blindly. It also hides real crawler activity if you block by user agent and accidentally block the real thing.
Blocking by user agent alone is weak, because the string is self-declared. Where the provider publishes IP ranges, block or allow by range instead. Keep a record of what you blocked and why, so a later review can check the decision.
The status code tells you what your server did with the request. A 200 means the page was served. A 304 means the crawler already had a fresh copy. A 403 or 429 means your server or CDN refused it. A 5xx means your server failed, and the crawler may retry.
Robots.txt rules are requests, not enforcement. RFC 9309, the Robots Exclusion Protocol, defines how crawlers should read and honour robots.txt. A compliant crawler that is disallowed may still fetch the file itself, and some crawlers fetch a page before reading the rules. So a log line showing a fetch of a disallowed path is not automatically a violation; check the timing and the crawler's documented behaviour.
Also check for repeated 5xx responses to a known crawler. That is a server problem, and it can quietly starve your site of crawler attention. It has nothing to do with your content strategy.
5. Building a repeatable workflow
A workflow is a fixed sequence you run on a schedule, so results are comparable week to week. Here is one that works with either approach.
- Rotate and archive the raw log weekly, so the source file is preserved.
- Filter candidate crawler lines by user agent string from current provider docs.
- Verify a random sample against published IP ranges or reverse DNS.
- Count verified hits per crawler per path, and note status codes.
- Log spoofed hits separately, with the IP and the claimed user agent.
- Store the weekly summary in one sheet, so trends are visible.
For the dashboard, four columns are enough: week, crawler name, verified hit count, and top requested path. Add a fifth column for spoofed hits. Anything more tends to go unread.
Server logs contain IP addresses, which count as personal data under the GDPR in the EU and UK. The UK Information Commissioner's Office publishes guidance on lawful log retention. Keep retention short, restrict access, and avoid joining logs to other datasets unless you have a lawful basis.
Sampling limits are real. If you analyse one day out of seven, you can describe that day. You cannot describe the month. State the sample in every report: "verified sample of N hits from week 12" is honest; "AI crawler traffic doubled" from the same data is not.
And the biggest limit: logs show requests, not outcomes. They cannot tell you whether a page was used in an AI answer or cited. There is no public figure for that conversion, because providers do not publish it. Any claim linking log hits to citations needs its own evidence.
Key Facts at a Glance
- Combined Log Format fields are documented in the Apache HTTP Server documentation.
- RFC 9309 defines how crawlers should honour robots.txt; it is a request, not enforcement.
- OpenAI and Anthropic publish crawler IP ranges; Google documents forward-confirmed reverse DNS.
- Behind a CDN, the logged client IP may be the proxy, so check the real-client header.
- Server logs contain IP addresses, which are personal data under GDPR; the ICO publishes retention guidance.
- Sampling limits mean a weekly sample describes a sample, not the month.
How do I identify AI crawlers in my server logs?
Filter log lines by the user agent strings listed in each provider's current documentation, then verify the client IP against the provider's published ranges. The string alone only narrows the list.
How can I verify a crawler's identity is real?
Check the client IP against the provider's published IP ranges, or run forward-confirmed reverse DNS where the provider documents that method. Do this on a random sample, not a single hit.
What do AI crawler hits actually tell me?
They tell you a request arrived and what your server returned. They do not tell you whether the page was used in an AI answer or cited. No public figure links log hits to citations.
Can I analyse server logs without a paid tool?
Yes. Standard shell tools can filter and count lines. A paid analyzer helps with scale and dashboards, but the verification step still needs the provider's published ranges.
How do I spot spoofed AI crawler traffic?
Look for hits that claim a major crawler but fail the IP check. Record the IP and the claimed string separately. Treat spoofing as a finding in its own right, not a rounding error.
