You can see which crawlers visit your site in two ways. Read the raw access log with command-line tools, or run a log analyzer that parses the file for you. The raw route costs nothing but your time. The analyzer route costs money or setup effort, but it handles volume and repeat checks. Which one fits depends on how often you check and how many hits you get.
An access log is a text file your web server writes for every request it receives. Each line is one request: who asked, what they asked for, when, and what the server sent back. If you can read that file, you can see crawler visits yourself, without trusting a third-party dashboard.
Your CMS dashboard cannot show you most crawler traffic
Most site owners check analytics and stop there. That misses crawlers that never run JavaScript. Analytics tools that rely on JavaScript tags cannot record a request from a bot that only fetches HTML. A page can be crawled weekly and still show zero sessions in your dashboard.
Server logs sit underneath all of that. They record the request whether or not a script ran. Google's documentation on verifying Googlebot tells you to check server logs when you need certainty about a request's origin. That makes logs the ground truth for crawl activity.
Analytics tells you about humans and script-running bots. Logs tell you about everything that hit the server.
Raw log reading and log analyzers solve different problems
Option A is the raw log read. You open the file, filter lines by user-agent or IP, and count. Option B is a log analyzer: a tool that parses the file and produces reports, usually with a crawler view built in.
Neither wins in the abstract. Raw reading wins when you have one specific question and a small file. An analyzer wins when you check monthly, when the file is large, or when more than one person needs the numbers.
| Factor | Raw log read | Log analyzer |
|---|---|---|
| Setup | None beyond server access | Install, or upload the file to a hosted tool |
| Skill needed | Basic command line | Reading a dashboard |
| Cost | Your time only | Free tiers exist; paid plans vary |
| Good for | One-off checks, small files | Monthly reporting, large files |
| Weakness | Manual, easy to miscount | Parsing rules can miss custom formats |
| Proof of identity | You run the check yourself | Depends on the tool's lookup |
That last row matters more than the rest. A tool that groups hits by user-agent string only tells you what the crawler claimed. It does not confirm the claim. You still need a verification step, and that step is the same in both options.
Find the log file, then read one line
On Apache, the access log is usually at /var/log/apache2/access.log or /var/log/httpd/access_log. On Nginx, look at /var/log/nginx/access.log. Managed hosts often expose logs in a control panel instead. If you cannot find the file, ask your host which path holds the raw access log. That is a normal support question.
A typical line follows the Common Log Format, which the Apache documentation defines. It looks like this:
203.0.113.10 - - [01/Oct/2026:09:14:02 +0000] "GET /pricing HTTP/1.1" 200 4512 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Read it left to right: IP address, timestamp, the request, the status code, the size, and the user-agent string. The user-agent is the crawler's self-declared name. It is a claim, not proof. Anyone can send a request that says it is Googlebot.
Verify a crawler's identity with two DNS lookups
Verification means confirming that the IP address sending the request actually belongs to the company that owns the crawler. The method is a reverse DNS lookup followed by a forward lookup.
- Step 1: take the IP from the log line.
- Step 2: run a reverse DNS lookup on that IP. For Googlebot, the hostname should end in
googlebot.comorgoogle.com. - Step 3: run a forward DNS lookup on that hostname. The result must match the original IP.
- Step 4: if the two match, the request is consistent with that crawler's published range. If they do not match, treat the request as unverified.
Google publishes this exact procedure in its documentation on verifying Googlebot. Microsoft's Bing documentation describes a similar check for Bingbot. OpenAI's documentation on its crawler, GPTBot, lists published IP ranges you can check against.
Ranges change, so re-check the vendor's page rather than a copied list.
Three things that look like AI crawler hits but are not
First, a hit does not mean the page was read into a model. It means the server delivered bytes. What happens after delivery is outside the log's view.
Second, a spoofed user-agent looks identical to a real one until you check the IP. Scrapers copy popular crawler strings because some sites allow them. A log line saying GPTBot proves nothing on its own.
Third, a 304 status code is still a visit. The server replied "not modified" and sent no body, but the crawler did make the request. Counting only 200 responses undercounts crawl activity.
For a broader view of how these crawlers differ, see our piece on training versus retrieval crawlers.
Four ways to read logs without a tool
You can answer most questions with a few commands. The examples below assume an Apache-style access log.
- Count hits per user-agent:
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20 - Find a specific crawler:
grep -i "gptbot" access.log | wc -l - See which paths a crawler fetched:
grep -i "gptbot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn - Check status codes for that crawler:
grep -i "gptbot" access.log | awk '{print $9}' | sort | uniq -c
Run these on a copy of the file, not the live one. Log files rotate, and a rotation mid-read can split your results. If the file is compressed, use zcat instead of cat.
One week of data is enough for a first look at patterns. Many hosts keep logs for 7 to 30 days, so download before they expire. Ask your host for its exact retention window.
Logs prove requests, not indexing or citations
Logs prove that a request arrived from a given IP with a given user-agent at a given time. That is a strong, checkable fact.
Logs do not prove that content was indexed, cited, or used in a model's answer. They also do not tell you why a crawler skipped a page. For that you need the crawler's own tooling, like Google Search Console's crawl stats report, which Google documents as a separate view from raw logs.
Treat logs as one input. Use them to confirm crawl behaviour, not to make claims about ranking or citation.
Handle logs as personal data
Access logs can contain personal data, because IP addresses can identify a person under rules like the GDPR in the EU and UK. Keep logs only as long as you need them, restrict who can read them, and avoid exporting raw lines into shared documents.
For crawler checks, you rarely need the full file. Filter to the crawler lines first, then share only those. This keeps the privacy surface small and the analysis focused.
Key Facts at a Glance
- Access log: a server-written text file, one line per request.
- User-agent: the crawler's self-declared name, a claim not proof.
- Verification: reverse DNS, then forward DNS, then compare the IP.
- Raw reading suits one-off checks; an analyzer suits repeat reporting.
- Logs show requests, not indexing or citation outcomes.
- IP addresses are personal data in the EU and UK; filter before sharing.
What can server logs tell you about AI crawlers?
They show which IPs requested which URLs, when, and with what user-agent string. They do not show what the crawler did with the content afterward.
How do you verify a crawler's identity?
Run a reverse DNS lookup on the IP, then a forward lookup on the resulting hostname. If the hostname resolves back to the same IP and matches the vendor's published range, the request is consistent with that crawler.
What tools work well for analyzing server logs?
Command-line tools like awk and grep work for small files. For repeat reporting, pick an analyzer that lets you define custom user-agent groups and export the raw lines, so you can still verify IPs yourself.
Can server logs show what content AI crawlers access?
Yes. The request path in each line shows which URLs were fetched. Counting paths per crawler shows which sections get attention.
How often should you analyze server logs for crawler activity?
Monthly is enough for most sites, plus an extra check after a major site change. If you are diagnosing a sudden traffic drop, check within days, because logs may rotate before you look.
If you have server access and one question, read the raw log. If you report monthly and the file is large, use an analyzer and keep the verification step manual. If a crawler hit looks suspicious, check the IP before you trust the name.