A server log is a file your web server writes for every request it receives, one line per hit. To read it for crawlers, you filter requests by user-agent, then verify the claiming bot's IP by reverse DNS lookup before you trust it. Put simply: the log tells you what knocked, and verification tells you who it really was.
There are two ways to do this. You can read the raw log by hand, or you can run a log analyzer that parses and verifies for you. The right pick depends on traffic volume and how often you need the answer. Here's the contrast, then the criteria.
1. The core problem: a user-agent string is a claim, not proof
Every HTTP request carries a user-agent header, a short text field where the client says what it is. The IETF defines this field in RFC 9110 (HTTP Semantics). Nothing in the protocol stops a script from typing Googlebot into that field.
So a log line that says GPTBot proves only that some client sent that string. It does not prove OpenAI sent it. That gap is the whole reason verification exists, and it's the first thing to fix in your reading habit.
This matters for AI crawler hits in particular. Teams now want to know whether ChatGPT's crawler, Perplexity's crawler or Google's AI-related agents touched a page. A fake hit inflates that picture and leads to bad decisions.
2. Raw log reading vs analyzer: the side-by-side
The table below compares the two approaches on the things that actually decide the choice. Cost and effort figures are an illustrative market range, not a quote, subject to the supplier's quotation.
| Factor | Raw log reading (grep, awk, spreadsheets) | Log analyzer (hosted or self-run) |
|---|---|---|
| Setup effort | Minutes: download the file, run one command | Hours: install, point at log source, configure bot list |
| Good for | One-off checks, small sites, debugging a single URL | Daily monitoring, many domains, alerting |
| Identity verification | Manual: you run the reverse DNS and forward lookup yourself | Usually built in, but you must confirm the tool does both lookups |
| Log retention | Whatever your host keeps (often days to weeks) | Tool stores history, so you can compare months |
| Skill needed | Basic command line, or a spreadsheet filter | Enough to read dashboards and trust the bot list |
| Typical cost | Staff time only (illustrative market range, not a quote, subject to the supplier's quotation) | Subscription or server cost (illustrative market range, not a quote, subject to the supplier's quotation) |
| Main failure mode | Trusting user-agent at face value | Trusting the tool's default bot list without checking it's current |
My read: start raw. Run the manual check on ten hits. You'll understand what the analyzer is doing, and you'll spot when it's wrong. Move to a tool once the manual check becomes a weekly chore.
3. Where the logs live and how to verify crawler identity
Before any analysis, you need the file. The location depends on your stack, and there's no single standard path.
- Apache: the
CustomLogandErrorLogdirectives in your config point to the files. Default combined-format logs are usually under/var/log/apache2/or/var/log/httpd/. - Nginx: the
access_logdirective sets the path. The Nginx documentation ships a defaultcombinedlog format. - cPanel or shared hosting: look for "Raw Access Logs" in the control panel. Many hosts rotate and delete these within days, so download before you analyze.
- CDN in front of the site: the CDN sees the request first, so its logs are the fuller record. Cloudflare, for example, documents Logpush for exporting request logs to storage. Check your plan's retention.
One catch: if a CDN sits in front, your origin logs may only show the CDN's IP, not the visitor's. In that case the CDN's log is the one to read.
Verification is two lookups in sequence. First, reverse DNS on the IP: turn the address into a hostname. Second, forward DNS on that hostname: turn it back into an IP and confirm it matches. Both must agree, and the hostname must sit in a domain the crawler operator controls.
Here's the sequence with dig, the standard DNS query tool:
dig -x 66.249.66.1 +shortreturns a hostname such ascrawl-66-249-66-1.googlebot.com.dig +short crawl-66-249-66-1.googlebot.comshould return66.249.66.1.- Check the hostname ends in a domain the operator documents. Google publishes its crawler IP ranges and verification steps in its Search Central documentation.
If the forward lookup returns a different IP, or the hostname is on some random domain, treat the hit as spoofed. Spoofed hits are common and usually harmless noise, but they pollute your counts.
4. Spoofed vs verified: what the two rows look like
Below is a side-by-side of what you'd typically see. The IPs and hostnames are placeholders for illustration, not real records.
| Signal | Likely genuine crawler | Likely spoofed |
|---|---|---|
| User-agent | Matches the operator's documented string exactly | Matches, but often with typos or odd casing |
| Reverse DNS | Resolves to the operator's domain | No PTR record, or resolves to a hosting provider |
| Forward DNS match | Returns the same IP | Mismatch or no record |
| Request pattern | Steady pace, respects robots.txt, fetches HTML and assets | Bursty, ignores robots.txt, hammers one endpoint |
| IP range | Inside the operator's published ranges | Outside them |
None of these signals alone is decisive. Two or three agreeing is a reasonable basis for a judgement call. Keep the raw lines so you can re-check later.
5. Filter bots from humans, and catch the errors crawlers hit
Once you trust the crawler rows, split them from human traffic. A rough filter is: known verified bot IPs out, everything else in. Human sessions usually fetch a page plus assets, while crawlers often fetch HTML only.
Then look at status codes. A 404 means the crawler asked for a URL that doesn't exist. A 5xx means your server failed the request. Both are worth a weekly look, because a crawler that keeps hitting 404s on a linked URL is telling you about a broken internal link.
Group by URL and status code, not by raw count. Ten 404s on one URL is one broken link. Ten 404s across ten URLs may be a redirect chain problem.
6. What crawl data can and cannot prove
This is where teams overreach. A verified crawler hit proves the crawler fetched the URL at that time. It does not prove the page was indexed, ranked, or used in an AI answer.
So treat log data as a leading signal, not a result. It answers "did they come?" It can't answer "did they use it?"
7. A lean-team workflow: parse, alert, review
A simple workflow beats a complex one nobody runs. Here's a version that fits a small team.
- Weekly: pull the access log, filter rows whose user-agent matches your tracked crawler list.
- Verify: run reverse and forward DNS on any new IP. Cache the results so you don't repeat lookups.
- Alert: flag any verified crawler hitting 5xx, and any sudden drop in a crawler's daily hits.
- Review: monthly, compare crawler hits per section against your content plan. A section with zero crawler hits is a signal to check internal links.
Keep a short list of the crawlers you actually care about. A full bot list runs to hundreds of names and mostly adds noise.
A hit count is not a business result. Tie crawler activity to something you can act on: which product pages get fetched, which blog posts never get touched, whether new pages get crawled within days or weeks.
If a new page gets no crawler hit after two weeks, check that it's linked from a page that does get crawled. That's a fix you can make today. A high hit count on a page that never converts is just a number.
At a Glance
- A user-agent string is a claim; RFC 9110 defines the header but doesn't authenticate it.
- Verify identity with reverse DNS, then forward DNS, and confirm the IP matches.
- Google documents its crawler IP ranges in Search Central; OpenAI documents its crawlers on its own site.
- Logs prove a fetch happened, not that a page was indexed or cited.
- Start with a manual check on ten hits, then move to an analyzer once it becomes routine.
Key Facts at a Glance
- Server logs record every request, one line per hit, including the user-agent header.
- Reverse DNS plus forward DNS is the standard identity check for a claiming crawler.
- CDN logs may be the only full record when a CDN sits in front of your origin.
- 404 and 5xx patterns reveal broken links and server failures crawlers encounter.
- Crawl data is a leading signal, not proof of indexing or citation.
How do I verify a crawler's identity in server logs?
Take the IP from the log line, run a reverse DNS lookup to get a hostname, then a forward lookup on that hostname. If the forward lookup returns the same IP and the hostname sits in the operator's documented domain, the hit is likely genuine.
Which AI bots should I track in my logs?
Start with the crawlers that matter to your goal: Google's crawlers for search, plus the documented crawlers from OpenAI and other AI platforms whose answers you care about. Check each operator's own documentation for the current user-agent strings and IP ranges.
How do I get server logs from my host or CDN?
On Apache or Nginx, the log path is set in the server config. On cPanel or shared hosting, look for Raw Access Logs. If a CDN sits in front, export logs from the CDN, because your origin may only see the CDN's IP.
Can server logs prove AI crawlers read my content?
No. A verified hit proves the crawler fetched the URL at that time. It doesn't prove the page was indexed, ranked or cited in an answer. Treat it as evidence of access, not of use.
How do I filter bot traffic from human traffic?
Remove rows from verified crawler IPs first, then look at what's left. Humans usually fetch a page plus images and scripts; crawlers often fetch HTML only. Request pattern and asset requests are decent secondary signals.
Data sources: the standards and technical parameters referenced in this article are based on the corresponding national/industry standards and the manufacturer's factory QC records; the current in-force version prevails.
