You can read server logs to see which crawlers visit your site in two ways: parse the raw access log yourself, or run it through a log analyzer tool. Raw parsing costs nothing and gives you full control. A tool is faster and easier on big sites. Neither one tells you the truth until you check crawler identity, because any script can copy a user-agent string. This piece compares both methods, then gives you a checklist you can run today.
Every bot can claim to be Googlebot, so the user-agent proves nothing
A server log records every request to your site. Each line usually holds the IP address, the date and time, the request method, the URL path, the HTTP status code, the response size, and the user-agent string. The user-agent is the text a client sends to say who it is, such as Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html).
That string is not a password. Any script can send it. A log line that says Googlebot might come from Google, or from a scraper in a data centre. Google's own guide, "Verify Googlebot", says to confirm the request comes from Google's published IP ranges, not from the name in the string.
AI crawlers have the same problem. OpenAI publishes IP ranges for its crawlers in JSON files linked from its documentation. Anthropic publishes ranges for ClaudeBot in its docs. Read only the user-agent and you can't tell a real AI crawler hit from a fake one. Both methods below have to solve this.
Raw analysis and log analyzers split on cost, control and skill
Raw log analysis means you write queries or scripts against the log file. A log analyzer is software that parses logs and shows dashboards. The table compares them on what matters for crawler checks.
| Factor | Raw log analysis | Log analyzer tool |
|---|---|---|
| Setup effort | High: you need the log format and some filters | Low: upload a file or connect a source |
| Cost | Free with tools you already have (awk, grep, Python) | Free tiers exist; paid plans for large volumes |
| Control over verification | Full: you script IP checks against published ranges | Varies; some tools take custom bot lists |
| Large files | Manual; you may need to split or stream | Built for big files; often samples or indexes |
| Learning curve | Steep without regex or SQL | Gentle: point and click |
| Good for | Small sites, one-off audits, custom checks | Ongoing monitoring, teams without log skills |
Neither method wins everywhere. Your site size, your team's skills and how often you check decide it. A site with 500 requests a day can be audited with a few command-line lines. A site with millions of requests needs a tool or a proper log pipeline.
A log line shows the request, not whether the crawler is real or the page got indexed
A typical access log line in Combined Log Format looks like this:
203.0.113.5 - - [10/Oct/2025:14:32:01 +0000] "GET /blog/post HTTP/1.1" 200 5123 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
The fields are: IP address, ident, authuser, timestamp, request line, status code, bytes sent, referrer, and user-agent. The Apache HTTP Server documentation defines this format, and Nginx and other servers use it too.
The log does not tell you whether the crawler is verified. It does not tell you if JavaScript rendered the page. It does not tell you if the content was indexed. Those are separate steps. Cross-check with other data, such as Google Search Console's Crawl Stats report.
For AI crawlers, the log shows a hit, but not whether the content was used in an answer. That needs the AI platform's own documentation or watching for citations, which sits outside the log.
Verification means matching the request IP to the operator's published range
Verification means checking the request's IP address against the official list the crawler operator publishes. Google publishes its ranges in its "Verify Googlebot" guide at https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot. OpenAI publishes its ranges in JSON files linked from its documentation. Anthropic publishes ClaudeBot ranges in its docs.
Here is a simple procedure you can run on any log file:
- Pull every line where the user-agent contains a known crawler token (e.g.,
Googlebot,GPTBot,ClaudeBot). - Take the IP address from each line.
- Check that IP against the published range for that crawler. Use a script, or a manual lookup for small samples.
- Mark lines that fail the check as unverified. They may be fake or misconfigured.
If you use a log analyzer, check whether it does this step. Many tools only group by user-agent, which is not enough. Look for a setting like "verify bots" or a custom bot list feature.
No public figure exists for how many fake crawler hits a typical site gets. It depends on your niche and traffic. The check is cheap, so run it before you trust any bot report.
Six questions turn a raw log into a crawler report
Use this checklist whether you analyze raw logs or use a tool. Each item is a question you can answer from the log data plus the verification step.
- Which crawlers appear? List unique verified crawler names and their request counts.
- Are AI crawlers present? Look for tokens like
GPTBot,ClaudeBot,PerplexityBot,Google-Extended. Verify each one. - What paths do they request? Group by URL path. This shows which content attracts crawlers.
- What status codes do they get? 200 means success. 403 or 429 means blocked or rate-limited. 404 means missing.
- When do they visit? Look at timestamps. A sudden spike may follow a sitemap update.
- Do they follow robots.txt? If a crawler requests a disallowed path, it may be ignoring your rules.
This checklist works for any site. If you use a tool, make sure it can export the raw lines behind each summary, so you can run the verification yourself.
Pick raw analysis for small sites and a tool for big ones or teams without log skills
Under 10,000 requests a day, with some comfort on the command line, start with raw log analysis. Use grep and awk to filter and count. For example, grep -i "GPTBot" access.log | wc -l counts GPTBot hits. Then pull the IPs and check them against OpenAI's published ranges.
On a large site, or with a team that has no log skills, use a log analyzer. Pick one that supports bot verification or lets you import custom IP ranges. Some tools ship built-in lists for major crawlers, but check the date on that list; crawler ranges change.
For ongoing monitoring, set up a scheduled job that runs the raw analysis and emails you a summary. You get the control of raw analysis with the convenience of a tool. This depends on your server access and automation skills.
There is no single right answer. Verify identity, because unverified user-agent data is noise.
Six mistakes sink most crawler log reads
Mistake one: trusting the user-agent. Always verify the IP. Mistake two: confusing crawler hits with indexing. A hit means a request; it does not mean the page was indexed or cited. Google's Crawl Stats documentation explains this distinction.
Mistake three: ignoring status codes. A crawler that gets 403 is not reading your content. Check your firewall and robots.txt. Mistake four: assuming all AI crawlers behave the same. Training crawlers and retrieval crawlers may follow different rules and rates. OpenAI's documentation describes GPTBot for training and OAI-SearchBot for search, for example.
Mistake five: not checking the date of the IP range list. Crawler IPs change. Use the latest published list. Mistake six: analyzing a single day. Crawlers may visit weekly or monthly. Use at least a 30-day window if you can.
Key Facts at a Glance
- Two methods: raw log analysis (free, full control) vs. log analyzer tool (faster, less skill needed).
- Always verify crawler identity by checking the IP against the operator's published range; the user-agent alone is not proof.
- Key fields: IP, timestamp, request line, status code, user-agent. Apache documents Combined Log Format.
- Checklist: list verified crawlers, AI crawler tokens, paths, status codes, visit times, robots.txt compliance.
- No public figure exists for the share of fake crawler hits; the check is cheap, so run it.
- Log hits do not equal indexing or citation; cross-check with Search Console or platform docs.
Can I see AI crawler hits in Google Analytics?
Generally no. Google Analytics usually does not track crawler requests because crawlers do not execute JavaScript. You need server logs or a tool that reads logs. Some CDN providers show bot traffic in their dashboards, but check whether they verify bots.
What is the difference between a crawler hit and a visit?
A hit is a single request for a file. A crawler visit may include many hits. Logs record hits. To count visits, group by IP and time window, but be careful: crawlers may use multiple IPs.
How do I verify Googlebot?
Use the IP ranges Google publishes in its "Verify Googlebot" guide at https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot. You can also do a reverse DNS lookup, but Google recommends IP verification. The guide gives the exact steps.
Do I need to verify AI crawlers like GPTBot?
Yes, if you want to trust the data. OpenAI publishes IP ranges for its crawlers. Check the IP against those ranges. Without verification, a hit could come from any script claiming to be GPTBot.
What log format should I use?
Combined Log Format is common and includes the user-agent and referrer. Nginx and Apache can both produce it. Check your server documentation for the exact configuration. If you use a CDN, the CDN may add fields like CF-Connecting-IP.