You can spot crawler visits in server logs in minutes. You can only trust them after a second, slower check. The quick pass reads the user-agent string. The verified pass confirms the request came from the crawler's published IP range. Most teams stop at the quick pass, then argue about numbers that were never real.

A server access log is a plain text file. The web server writes one line per request: who asked, what they asked for, when, and what it answered. Reading it well means knowing which of those fields you can trust.

This guide compares the two approaches side by side. Then it gives you a monthly workflow you can repeat.

1. Two ways to count a crawler produce two very different numbers

Say a report claims Googlebot fetched 4,200 pages last month. That number could come from a user-agent match or from a verified IP check. The first is a guess dressed as a fact. The second is evidence.

User-agent matching is fast and needs no setup. It also catches every script that types "Googlebot" into a header. Verification costs one lookup per request batch, but it survives a spoofed header.

The choice matters most for AI crawlers. OpenAI, Anthropic and Google all publish the IP ranges their crawlers use, so verification is possible. A log line claiming to be GPTBot from an unrelated cloud host is not GPTBot.

2. An access log line carries nine fields, and only some are trustworthy

Most servers write logs in the Common Log Format or the Combined Log Format. The Combined format adds two fields to the common one: referrer and user-agent. A typical line looks like this:

203.0.113.10 - - [12/Mar/2026:09:14:02 +0000] "GET /pricing HTTP/1.1" 200 5120 "-" "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)"

The fields, in order: client IP, identity (usually empty), authenticated user (usually empty), timestamp, request line, status code, bytes sent, referrer, user-agent. The Apache HTTP Server documentation describes this format directly. nginx uses a compatible default.

FieldWhat it tells youHow much to trust it
Client IPThe address that opened the connectionHigh, unless a proxy or CDN sits in front
TimestampWhen the request arrivedHigh; check the timezone offset
Request lineThe URL and method requestedHigh
Status codeWhether the server served, redirected or refusedHigh
Bytes sentResponse size; 0 bytes on many 304sHigh
ReferrerWhere the request came fromLow for bots; crawlers rarely send one
User-agentThe name the client claimsLow until verified against the IP

Two things break this picture. If you use a CDN, the client IP in the log may be the CDN's edge node, not the crawler. Cloudflare's documentation explains how the real client address arrives in a header such as CF-Connecting-IP. You need to log that header to see it.

Log rotation also bites. Default setups often keep a few weeks of logs. If you want a monthly audit, confirm the retention window before you start. Otherwise the data is gone by the time you look.

3. User-agent matching is fast, rough, and easy to fake

User-agent matching means filtering log lines where the user-agent field contains a known crawler token, such as GPTBot, ClaudeBot or Googlebot. It is one command and it works on any log file.

With a standard Unix toolchain, a filter looks like this: grep -i "GPTBot" access.log | wc -l. That counts GPTBot lines. Swap the token for another crawler name to count that one. A log analyzer tool does the same job with a dashboard.

The weakness is real. Anyone can send a user-agent string. A scraper that wants to look like a search crawler sets the header and moves on. Your count goes up, and your conclusions go wrong. Google's documentation on verifying Googlebot says user-agent alone is not enough for exactly this reason.

Use this option for a first glance, a rough trend, or a sanity check on a site with no known scraping issues. Do not use it to prove a crawler visited.

4. Reverse DNS and IP-range checks turn a claim into evidence

Verification means checking the client IP against the range the crawler operator publishes, or resolving the IP to a hostname that belongs to that operator. Google publishes its ranges in a JSON file at googlebot.json on developers.google.com. OpenAI publishes GPTBot ranges in its own documentation. Anthropic documents ClaudeBot's IP ranges in its support pages.

A common check combines two steps. First, a reverse DNS lookup on the IP: dig -x 203.0.113.10 +short or host 203.0.113.10. Second, a forward lookup on the returned hostname to confirm it points back to the same IP. Google's verification documentation describes this forward-confirmed reverse DNS method.

The IP-range method is simpler when the operator publishes a list. Download the list, then test each logged IP against the ranges with a small script or your log analyzer's built-in filter. A match is strong evidence. A miss on a line that claims to be that crawler is a spoof signal.

CheckUser-agent matchVerified IP / reverse DNS
Setup effortOne filter lineRange list download, or a DNS lookup
SpeedInstant on any log sizeSeconds per IP; cache results
Resists spoofingNoYes, if the range list is current
Works for AI crawlersNames onlyYes, where ranges are published
Strong useQuick trend, first lookReports, audits, citation decisions

One limit: not every crawler publishes a range, and some publish ranges that change. Check the operator's documentation each quarter. If a crawler has no public range, you can only report a user-agent count and label it as unverified. That is an honest answer.

5. Three signals separate a spoofed crawler from a real one

The first signal is a user-agent that claims a crawler but resolves to an unrelated host. A reverse lookup returning a cloud provider's generic hostname, not the operator's domain, is a warning. Confirm with forward DNS before you conclude anything.

The second signal is behaviour. Real crawlers usually request pages at a steady, modest rate and follow links. A script hammering one endpoint hundreds of times a minute with no referrer and no variation is not a normal crawl pattern.

The third signal is the request mix. Crawlers fetch HTML, and sometimes assets. A client that only ever requests a JSON API endpoint while claiming to be a search crawler is worth a closer look. None of these signals proves intent. They justify a verification step.

6. Comparing sitemap URLs to verified hits finds pages crawlers never reach

To separate bots from people, start from the verified list and subtract. Everything left is either a human, an unverified bot, or monitoring traffic. Check the status codes on the human side: a burst of 404s from human IPs usually means broken links, not visitors.

Finding pages crawlers cannot reach is the more useful question. Take your sitemap URL list and compare it to the verified crawler requests in the log. URLs in the sitemap with zero verified hits over a month are candidates for a problem: blocked by robots.txt, behind a redirect chain, or simply not linked.

Then check status codes on the crawler side. A page that returns 403 or 503 to a verified crawler is not being read. A page that returns 200 to a crawler but never appears in an index is a separate issue. Logs show the fetch, not the indexing decision. Google's documentation on how Search works makes that distinction clear.

7. A repeatable monthly audit takes five steps

Step one: export the month's logs, including any CDN client-IP header you log. Confirm the timezone offset so the month boundary is right.

Step two: count user-agent matches per crawler. This is your rough number and your first table column.

Step three: verify each crawler's IPs against the published range list, or run forward-confirmed reverse DNS. Record the verified count and the mismatch count.

Step four: list sitemap URLs with zero verified crawler hits, and list URLs that returned 4xx or 5xx to verified crawlers. These are your action items.

Step five: store the results with the date and the range-list version you used. Next month you compare like with like. A free analyzer tool handles steps two and three; steps four and five need a spreadsheet or a small script.

8. Logs prove access, not indexing or citation

Logs can prove a request arrived from a given IP at a given time for a given URL. They can prove the server's response status. With verification, they can tie the request to a named crawler operator.

Logs cannot prove the page was indexed, ranked or cited by an AI answer. Crawling is a fetch. Indexing and citation are separate decisions made downstream. If a page was fetched by a crawler in March 2026 and never cited, the log tells you the fetch happened, nothing more.

Logs also cannot show you what a crawler did with the content. Treat the log as evidence of access, not evidence of outcome. For citation questions, you need a separate check of the answer engine itself.

Key Facts at a Glance

  • Access logs record IP, timestamp, URL, status and user-agent per request; the Combined Log Format adds referrer and user-agent.
  • User-agent matching is fast but spoofable; verify against the operator's published IP ranges for any report.
  • Google, OpenAI and Anthropic publish crawler IP ranges you can check against.
  • Forward-confirmed reverse DNS is the standard verification method for crawlers without a published range.
  • Zero verified crawler hits on a sitemap URL is a signal to check robots.txt, redirects and status codes.
  • Logs prove access, not indexing or citation.
How do I identify AI crawlers in my server logs?

Filter the user-agent field for known crawler tokens such as GPTBot or ClaudeBot. Then verify each matching IP against the operator's published range list or with a reverse DNS lookup. The user-agent gives you a name; the IP check gives you evidence.

How can I verify a crawler's identity beyond user-agent?

Use forward-confirmed reverse DNS, or compare the client IP against the operator's published range list. Google documents both methods for Googlebot. OpenAI and Anthropic publish their crawler ranges in their own documentation. A mismatch between claimed name and verified IP is a spoof signal.

What can server logs not tell you about crawlers?

Logs show that a fetch happened and what the server returned. They do not show whether the page was indexed, ranked or cited in an AI answer. Those are separate decisions made by the platform after the crawl.

How do I find pages crawlers cannot reach?

Compare your sitemap URL list to verified crawler requests in the log. URLs with zero verified hits over a month are candidates. Check their status codes for verified crawlers: 403, 503 or a long redirect chain will stop a fetch.

Which log analyzer tools handle AI crawler traffic?

Most general log analyzers filter by user-agent and let you import a custom IP list, which is enough for the verified pass. The specific tool matters less than whether it can join a log line to a published range list and export the result. Check the tool's own documentation for custom-list support before you commit.