You want to know which crawlers actually hit your site. Reading server logs for crawlers means matching each log line to a known user agent, then checking the IP address against the crawler operator's published list. This article compares two ways to do it: manual grep commands and a log analyzer tool. The right choice depends on how many lines you have and how often you check.
A server access log is a plain text file where the web server records one line per request. That line holds the IP address, the date, the path requested, the status code, and the user agent string. The user agent is a text label the client sends to say who it is. Anyone can copy it, so a label alone proves nothing.
1. Your log is noisy and labels lie
Most sites get far more bot traffic than human traffic. A busy site can produce hundreds of thousands of lines a day. Searching that by eye is hopeless, so you need a method.
The bigger problem is spoofing. Any script can send User-Agent: GPTBot and look like OpenAI's crawler. OpenAI publishes its IP ranges so you can check. Google publishes a JSON file of its crawler IPs. Cloudflare documents a similar verification approach for its own bot management. Without an IP check, a hit count is just a claim.
So the real question is not "which user agent strings appear in my log" but "which requests came from the operator's real network". Both approaches below answer that, but at different cost.
2. Manual grep and a lookup table work for one-off audits
Manual analysis means running text search commands against the raw log file. It is free, it works on any server, and you see every line. It scales badly past a few million lines.
Start by counting user agent hits. This gives a rough map before you verify anything:
grep -c 'GPTBot' access.log
grep -c 'ClaudeBot' access.log
grep -c 'PerplexityBot' access.log
grep -c 'Googlebot' access.log
Next, pull the IPs behind one agent and compare them to the operator's published range. For GPTBot, OpenAI lists its ranges at openai.com/gptbot.json. For Googlebot, the list is at developers.google.com/search/apis/ipranges/googlebot.json. A quick sort and count shows whether the IPs cluster inside those ranges or spread across random hosting providers:
grep 'GPTBot' access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
If most IPs sit outside the published range, you are looking at spoofed traffic. That is a finding in itself. The manual method is good for a one-off audit, a spot check after a traffic spike, or a server where you cannot install tools.
3. A log analyzer tool pays off when you check often
A log analyzer is software that parses log files, groups requests by user agent, IP and status code, and draws charts. Examples include GoAccess, which runs in a terminal and handles common log formats, and Matomo or AWStats, which run on a server and keep history.
The advantage is repeatability. Set the analyzer to ingest logs nightly and you get a crawler dashboard without typing commands. The disadvantage is setup time, storage, and the risk that a default report hides what you care about. Most analyzers group by user agent, not by verified IP range, so spoofed hits still show up as GPTBot unless you add a filter.
GoAccess is the lightest option for a single server: it reads the log directly and renders a report in the browser. For multi-server setups, a hosted analyzer or a small script that ships logs to one place works better. Either way, check whether the tool lets you import the operator IP list. If it does not, keep the manual IP check as a separate step.
4. Side-by-side comparison
| Factor | Manual grep | Log analyzer tool |
|---|---|---|
| Setup time | Minutes; no install | Hours to a day, plus config |
| Cost | Free | Free to paid depending on tool |
| Good for | One-off audits, spot checks | Ongoing monitoring, trend charts |
| Handles spoofing | Yes, if you add the IP check | Only if the tool supports IP filters |
| Scale limit | Slows past a few million lines | Designed for large volumes |
| Skill needed | Basic command line | Install and config knowledge |
| History | None unless you save output | Keeps history by default |
Neither option is "better" in the abstract. The table shows the trade: manual is fast to start and slow to repeat; the tool is slow to start and fast to repeat. Pick by how often you need the answer.
5. Both options need the same six verification checks
Whatever method you use, run these six checks. They turn a hit count into a verified finding.
- Check the user agent string. Confirm the label matches the crawler you expect, e.g.
GPTBot/1.2orClaudeBot/1.0. Version numbers change; treat the name as the key. - Resolve the IP to the operator's published range. Use the operator's own JSON or documentation. Do not trust a third-party list that may be out of date.
- Run a reverse DNS lookup where the operator supports it. Google documents that Googlebot IPs resolve to googlebot.com or google.com. A forward-confirmed reverse DNS check confirms the name matches the IP.
- Check the status codes. A wall of 403 or 429 responses means the crawler is being blocked or rate-limited, not that it is reading your content.
- Look at the paths requested. Crawlers that fetch only the homepage are not indexing deep pages. Compare paths to your sitemap.
- Compare before and after a change. If you edited robots.txt or added a block rule, compare the crawl log for the week before and the week after. That is the only clean test.
On the reverse DNS point: not every operator publishes a DNS pattern. OpenAI's documentation focuses on IP ranges, so the IP check is the primary control there. Google publishes both ranges and DNS names. Use whatever the operator documents, and mark the rest as unverified.
6. Three metrics carry most of the signal
Hit count shows how often a crawler visits. Status code mix shows whether it can actually read the page. Crawl frequency shows whether coverage is growing or flat.
A high hit count with mostly 200 responses means the crawler is reading. A high count with mostly 403 responses means your block rule is working, or your firewall is stopping a crawler you wanted. A low count on a new site is normal; expect crawl activity to grow over the first weeks as the site earns links and the crawler's scheduler revisits.
There is no public benchmark for "good" AI crawler hit volume. It depends on site size, update rate and how many operators know about you. Treat your own month-on-month trend as the baseline, not an industry figure.
7. If X, then Y: the decision summary
If you check logs once a quarter, use manual grep and the IP list. The setup cost of a tool will not pay back.
If you run a site with daily publishing and want to see crawl trends, install GoAccess or a hosted analyzer, then add the operator IP filter as a separate step.
If you see spoofed GPTBot or ClaudeBot traffic from random IPs, do not block the real crawler. Block the spoofed IPs at the firewall and keep the verified ranges open.
If you want to stop a specific AI crawler, decide between robots.txt and server-level blocking. Robots.txt is a request; a firewall rule is enforcement. Google's documentation notes that Googlebot honours robots.txt, and OpenAI documents that GPTBot respects robots.txt. Server-level blocking catches crawlers that ignore the file, but it also blocks any tool that shares the IP range.
Key Facts at a Glance
- Definition: a server access log is a text file recording one line per HTTP request, including IP, path, status and user agent.
- Two methods: manual grep is fast to start; a log analyzer is fast to repeat.
- Verification is not optional: user agent strings can be spoofed, so check the IP against the operator's published list.
- Key operators to check: OpenAI (GPTBot), Anthropic (ClaudeBot), Perplexity (PerplexityBot), Google (Googlebot). Each publishes its own IP documentation.
- Strong first step: count hits per user agent, then verify the top one against its published IP range.
- Decision rule: block spoofed IPs, not the real crawler range; use robots.txt for policy and firewall rules for enforcement.
Frequently Asked Questions
How do I identify AI crawlers in my server logs?
Search the log for known user agent names such as GPTBot, ClaudeBot and PerplexityBot. Then check the IP addresses against the operator's published range. A name alone is not proof.
How can I verify a crawler is legitimate and not spoofed?
Compare the request IP to the operator's published IP list. For Googlebot, also run a reverse DNS lookup and confirm the hostname resolves back to the same IP. If the IP is outside the published range, treat it as spoofed.
What is the difference between GPTBot and other AI crawlers?
They are run by different companies with different crawl policies. OpenAI documents GPTBot and its IP ranges on its own site. Anthropic and Perplexity publish their own documentation. Check each operator's page for current details, because names and ranges change.
Should I block AI crawlers in my server logs?
That depends on your goal. Blocking stops the crawler from reading your pages, which also stops any citation from that engine. If you want citations, allow verified crawlers and block spoofed IPs instead. Use robots.txt for policy and a firewall rule for enforcement.
Which log analyzer tools work well for crawler analysis?
GoAccess is a common free choice for a single server because it runs in the terminal. Matomo and AWStats keep history across days. None of them verify IP ranges by default, so plan a separate IP check.
Data sources: the standards and technical parameters referenced in this article are based on the corresponding national/industry standards and the manufacturer's factory QC records; the current in-force version prevails.
