You can find out which crawlers hit your site in about ten minutes. Open the raw access log, filter the lines that match known bot user-agents, then verify the ones that matter with a reverse DNS lookup. The user-agent string alone proves nothing, because any client can copy it. This guide compares raw log reading with a log analyzer, and gives you a checklist you can run this week.
1. An access log writes one line per request, and the user-agent field is the one that names the crawler
An access log is a plain text file. Your web server appends one line every time something requests a page. On Apache and Nginx the default format is the Common Log Format, described in the W3C's Logging Control In W3C httpd note and in Apache's own LogFormat documentation. A typical line looks like this:
203.0.113.42 - - [14/May/2026:09:12:03 +0000] "GET /pricing HTTP/1.1" 200 18432 "-" "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)"
The fields are: client IP, identity, authenticated user, timestamp, request line, status code, response size, referrer, and user-agent. If your log has only six fields, you are on the older Common Log Format and you are missing the referrer and user-agent. Fix that first, because you cannot do crawler work without the user-agent.
Where the file lives depends on the server. On Nginx it is usually /var/log/nginx/access.log. On Apache, /var/log/apache2/access.log on Debian-family systems or /var/log/httpd/access_log on Red Hat-family systems. On a managed host, look in the control panel's "Raw Access Logs" or "Access Logs" section.
2. Raw reading wins for verification, an analyzer wins for repeatable weekly reports
You have two approaches. You can read the raw log with command-line tools, or you can point a log analyzer at the file and let it group and chart the traffic. Both work. The choice depends on whether you need an answer today or a dashboard you can check every Monday.
Raw reading is fast, free, and gives you exact evidence. You see the actual IP and user-agent string, which is what you need for verification. The downside is that you have to type the same commands each time, and you have to know what the tokens look like.
A log analyzer, such as GoAccess or AWStats, parses the file and gives you tables, charts, and saved reports. The upside is repeatability. The downside is that most analyzers group by user-agent string only, which means a spoofed bot looks identical to a real one until you cross-check.
| Task | Raw log reading | Log analyzer |
|---|---|---|
| Time to first answer | Minutes | Minutes to an hour, depending on setup |
| Shows exact IP and user-agent | Yes, line by line | Usually yes, in a detail view |
| Grouping and charts | Manual | Built in |
| Repeatable weekly report | You script it or retype it | Save a config, rerun |
| Verification by reverse DNS | Straightforward with dig -x | Often needs a separate step |
| Works on very large logs | Yes, with grep and awk | Yes, but memory-heavy |
| Good for | Spot checks, verification, incident work | Ongoing monitoring and reporting |
3. Each AI crawler publishes its own user-agent token, and you grep for those tokens
A user-agent token is the short name a crawler puts in its user-agent string to identify itself. The tokens below are published in each operator's own documentation. Treat them as starting points, not a complete list.
- GPTBot - OpenAI's crawler; documented at
platform.openai.com/docs/gptbot - OAI-SearchBot - OpenAI's search crawler; same documentation page
- ChatGPT-User - fired when a ChatGPT user asks it to open a page; same documentation page
- ClaudeBot and Claude-Web - Anthropic crawlers; documented at
support.anthropic.com - PerplexityBot - Perplexity's crawler; documented at
perplexity.ai/perplexitybot - Google-Extended - Google's token for AI training use; documented at
developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - Googlebot - Google's main search crawler; same documentation page
- Bingbot - Microsoft's search crawler; documented at
bing.com/webmasters - Applebot and Applebot-Extended - Apple's crawlers; documented at
support.apple.com - CCBot - Common Crawl; documented at
commoncrawl.org/ccbot
To count hits per token on a Linux server, run one command per token. For example: grep -c "GPTBot" /var/log/nginx/access.log. To see the top URLs each one fetched, pipe through awk and sort: grep "GPTBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20.
4. RFC 9110 says User-Agent is informational, so a copied string is not proof of identity
Any HTTP client can set any user-agent string it likes. The HTTP/1.1 specification, RFC 9110 section 10.1.5, treats User-Agent as informational, not as an identity credential. So a scraper that wants to look like GPTBot can simply copy the string.
That is why verification matters. The standard method, used by Google and documented in its Verifying Googlebot guide, is reverse DNS. You take the client IP address, run a reverse lookup, and check that the resulting hostname belongs to the operator's published domain. Then forward-resolve that hostname and confirm it points back to the same IP.
OpenAI publishes a similar method for GPTBot, listing the IP ranges it uses. Anthropic and Perplexity publish equivalent pages. Always check the operator's own documentation rather than a third-party list, because IP ranges change.
5. Five checks in order will tell you whether a crawler hit is real
Run these steps in order. They take about ten minutes for a normal site.
- Isolate the line. Grep the log for the token and copy the client IP.
- Reverse-resolve the IP. Run
dig -x 203.0.113.42 +short. You should get a hostname ending in the operator's domain, such ascrawl-203-0-113-42.openai.com. - Forward-resolve the hostname. Run
dig crawl-203-0-113-42.openai.com +short. The answer must match the original IP. - Check the published range. Compare the IP against the operator's official range list.
- Log the result. Record date, token, IP, hostname, and pass or fail in a spreadsheet.
If any of the three checks fails, treat the hit as unverified. It may be a spoofed bot, a shared proxy, or a CDN edge. Do not block on one bad line, but do count it separately from verified hits.
6. Compare what robots.txt allows with what the log shows, and read the status codes
Robots.txt is a file at the root of your site that tells crawlers which paths they may fetch. It is a request, not a lock. RFC 9309 defines the rules. If you disallow a path and a crawler still requests it, that is worth noting in your log review.
Status codes tell you what happened. A 200 means the page was served. A 304 means the crawler used its cached copy. A 403 or 429 means you or your server blocked or throttled the request. A 5xx means your server failed, and the crawler may retry.
For each verified crawler, pull the status-code mix. If a real AI crawler is getting 403 on the pages you want cited, your firewall or CDN rule is likely blocking it. That is a fixable problem, and the log is where you see it.
7. A fixed weekly routine turns a snapshot into a trend line
One-off log reading tells you what happened last month. A weekly routine tells you what is changing. Pick a fixed day, such as Monday morning, and run the same four queries: total hits per verified token, top ten URLs per token, status-code mix, and new IP ranges that were not in last week's list.
Keep the output in one file per week. After eight weeks you have a trend line, which is more useful than any single snapshot. You can also see whether a new AI crawler has appeared before it shows up in a third-party list.
If you want to automate the routine, a small script that runs the four queries and emails the output is enough. You do not need a commercial platform for a site of normal size.
8. Access logs hold personal data when they store IP addresses, so set a retention window
Under the UK GDPR and the EU GDPR, an IP address that can identify a person is personal data. The UK Information Commissioner's Office publishes guidance on personal data and identifiers, and the European Data Protection Board publishes its own guidance. Check your own jurisdiction and take legal advice on the retention period that applies to you.
Practically, that means three things. Set a retention window and delete older logs. Configure log rotation so files do not grow without limit. And if you export log data into an analytics tool, check whether the export strips or hashes the IP field.
A common setup is 30 to 90 days of raw logs, then aggregated counts kept longer with IPs removed. The right number depends on your legal advice, not on a blog post.
Key Facts at a Glance
- Access logs record one line per request; the user-agent field is what identifies a crawler.
- Raw reading is good for verification; a log analyzer is good for weekly reporting.
- RFC 9110 treats User-Agent as informational, so verify the ones that matter by reverse DNS.
- Keep a fixed weekly routine: hits per token, top URLs, status mix, new IP ranges.
- Treat access logs as personal data where IPs are stored; set a retention window with legal advice.
How do I identify AI crawlers in my server logs?
Grep the log for known tokens such as GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, then count the matches. Each operator publishes its tokens in its own documentation.
Can I trust the user-agent string to verify a crawler?
No. RFC 9110 treats User-Agent as informational, so any client can copy it. Verify by reverse DNS: look up the IP, check the hostname belongs to the operator, then forward-resolve it back to the same IP.
What is the difference between robots.txt and server logs?
Robots.txt is a request to crawlers about which paths they may fetch, defined in RFC 9309. Server logs are a record of what was actually requested. Comparing the two shows whether crawlers are following your rules.
How often should I analyze my server logs?
Weekly is enough for most sites. Run the same four queries each time so you can compare week to week rather than starting from scratch.
What do AI crawler hits actually prove?
They prove a request came from an IP claiming that user-agent. If the IP passes reverse DNS and range checks, you can treat it as a verified hit. They do not prove the content was used in an answer.
Sources: W3C Logging Control In W3C httpd; Apache LogFormat documentation; RFC 9110 section 10.1.5; RFC 9309; Google Search Central Verifying Googlebot; OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple and Common Crawl crawler documentation pages; UK ICO and European Data Protection Board guidance on personal data.