Read Server Logs for Crawlers: Two Approaches

Q: How do I get server logs from my host or CDN?

On Apache, look in /var/log/apache2/ or /var/log/httpd/ . On Nginx, look in /var/log/nginx/ . On shared hosting, download raw access logs from cPanel. On a CDN, export logs from the dashboard or API, or use the provider's bot report.

You can read server logs to see which crawlers visit your site in two main ways. Use command-line tools like grep and awk if you want control and no extra cost. Use a log analyzer or CDN dashboard if you want speed and visual reports. The right choice depends on your team's skills, log volume and how often you need to check.

A server log is a text file where your web server records every request, including the visitor's IP address, the user-agent string and the response code. A crawler is a bot that fetches pages automatically, often for search engines or AI systems. Reading logs tells you who actually came, not who claimed to come.

1. Why crawler logs matter more than user-agent strings

User-agent strings are easy to fake. Any script can send a header that says "Googlebot" or "GPTBot". The log entry alone does not prove identity. You need to check the source IP address against the crawler's published ranges.

Google publishes its crawler IP ranges in a JSON file at googlebot.json under developers.google.com/search/apis/ipranges. OpenAI documents its crawler IPs at openai.com/gptbot.json. Cloudflare maintains a public list of verified bots at radar.cloudflare.com/traffic/verified-bots. These are the authorities to check.

If the IP does not match the published range, treat the hit as unverified. It might be a spoofed bot, a scraping tool or a misconfigured proxy. Log analysis without IP verification is just guessing.

2. Approach A: CLI filtering with grep and awk

The command-line approach uses tools already on most Linux servers. You filter log lines by user-agent, count hits and extract IPs. It costs nothing and works on any log size, though large files take time.

Start by finding your log file. Apache usually writes to /var/log/apache2/access.log or /var/log/httpd/access_log. Nginx uses /var/log/nginx/access.log. On shared hosting, download logs from cPanel under "Raw Access Logs". On a CDN, export logs from the dashboard or API.

Here are four commands to start with. Run them on a copy of the log to avoid locking the live file.

grep -i "GPTBot" access.log | wc -l counts hits from a user-agent containing GPTBot.
grep -i "Googlebot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn lists the top IPs claiming to be Googlebot.
grep -i "ClaudeBot" access.log | grep " 404 " shows 404 errors hit by ClaudeBot.
awk '{print $9}' access.log | sort | uniq -c | sort -rn counts response codes across all requests.

The first field in a combined log format is the IP address. The ninth field is the HTTP status code. That format is defined in the Apache documentation and followed by Nginx by default.

3. Approach B: log analyzers and CDN dashboards

A log analyzer is software that parses log files and produces charts, tables and summaries. Examples include GoAccess (open source), AWStats (open source) and commercial log SaaS platforms. CDN providers also offer bot reports in their dashboards.

GoAccess runs in a terminal and can show a live dashboard. AWStats produces static HTML reports. Commercial tools add alerts and retention. The trade-off is setup time and, for SaaS, cost.

CDN dashboards are the fastest route if your site sits behind Cloudflare, Akamai or Fastly. They often separate verified bots from unverified ones. Check the provider's documentation for how they classify traffic, because the labels differ.

4. Side-by-side comparison

The table below compares the two approaches on the factors that usually decide the choice.

Factor	CLI (grep/awk)	Log analyzer / CDN dashboard
Cost	Free with the server	Free (GoAccess) to paid SaaS
Setup time	Minutes if you know the commands	Minutes for CDN dashboard; longer for self-hosted analyzers
Skill needed	Basic shell and regex	Point-and-click; some tools need config files
Log size limit	Handles large files but slowly	Varies; SaaS may cap ingestion
Bot verification	Manual: you check IPs against published ranges	Often built in, but check the provider's method
Good for	One-off checks, custom filters, no budget	Ongoing monitoring, team reports, non-technical users

5. A self-check list before you trust any crawler hit

Use this list every time you review logs. It takes five minutes and prevents most false conclusions.

Confirm the log covers the date range you care about. Check for gaps from rotation or sampling.
Extract the IP address and user-agent for each hit you plan to count.
Look up the IP in the crawler's published range: Google's JSON, OpenAI's JSON or Cloudflare's verified bots list.
Run a reverse DNS lookup with dig -x IP or host IP. Googlebot IPs resolve to googlebot.com or google.com. Then forward-resolve the hostname to confirm it points back to the same IP.
Check the response code. A 200 means the page was served. A 404 or 5xx means the crawler hit a problem.
Compare hits over time. A sudden spike from one IP often means a scraper, not a crawler.

Reverse DNS is not proof on its own, because an attacker can control DNS for their own IP. The forward-confirmed reverse DNS (FCrDNS) check is stronger: resolve the IP to a hostname, then resolve that hostname back to an IP and compare. Google's documentation recommends this method for Googlebot.

6. Spoofed versus verified: what the log lines look like

A verified Googlebot hit usually shows an IP inside Google's published ranges and a hostname ending in googlebot.com or google.com. A spoofed hit shows an unrelated IP, often from a cloud provider or residential range, with the same user-agent string.

Here is a simplified example. The first line is plausible; the second is not.

66.249.66.1 - - [10/Mar/2025:10:00:00 +0000] "GET /page HTTP/1.1" 200 512 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.0.113.45 - - [10/Mar/2025:10:01:00 +0000] "GET /page HTTP/1.1" 200 512 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The second IP is in a documentation range and does not belong to Google. In a real log, you would check the actual IP against the published list. Do not trust the user-agent alone.

7. Handling 404s and 5xx errors from crawlers

Crawlers report problems through status codes. A 404 means the URL does not exist. A 5xx means your server failed. Both can waste crawl budget and hide content from AI systems.

Filter for 4xx and 5xx hits by known crawlers. Fix broken internal links that point to 404s. Check server logs for 5xx bursts around the time a crawler visited; they often point to timeouts or resource limits.

For AI crawlers, a 403 or 429 may mean your robots.txt or rate limiting is blocking them. Review your rules before assuming the bot is broken.

8. Deciding whether to block an AI crawler

Blocking is a business decision, not a technical one. Training crawlers and retrieval crawlers serve different purposes. A training crawler collects content for model training. A retrieval crawler fetches pages to answer a query in real time.

OpenAI documents GPTBot as a training crawler and OAI-SearchBot as a search crawler, with separate IP ranges. Anthropic documents ClaudeBot and its other agents in its support pages. Check each vendor's current documentation before you change robots.txt.

If you want citations in AI answers, blocking retrieval crawlers works against that goal. If you want to limit training use, target training crawlers. Record the decision and the date, because vendor policies change.

Key Facts at a Glance

Two approaches: CLI filtering (grep/awk) for control and zero cost; log analyzers or CDN dashboards for speed and reports.
Always verify crawler IPs against the vendor's published range: Google's JSON, OpenAI's JSON or Cloudflare's verified bots list.
Reverse DNS plus forward confirmation is the recommended check for Googlebot, per Google's documentation.
Log retention and sampling affect accuracy; check for gaps before drawing conclusions.
Blocking training crawlers and blocking retrieval crawlers have different effects on AI citation.

How do I verify a crawler's identity in server logs?

Extract the IP address from the log line, then check it against the crawler vendor's published IP range. For Googlebot, use Google's JSON file. Run a reverse DNS lookup and confirm the hostname resolves back to the same IP. If either check fails, treat the hit as unverified.

What do AI crawler hits look like in access logs?

They look like any other request, with a user-agent string naming the crawler, such as GPTBot or ClaudeBot. The difference is the source IP, which should fall inside the vendor's published range. The response code tells you whether the page was served or blocked.

Should I block AI crawlers from my site?

It depends on your goal. Blocking training crawlers limits use of your content for model training. Blocking retrieval crawlers can reduce your chances of being cited in AI answers. Check each vendor's current documentation and decide per crawler, not as a blanket rule.

How do I get server logs from my host or CDN?

On Apache, look in /var/log/apache2/ or /var/log/httpd/. On Nginx, look in /var/log/nginx/. On shared hosting, download raw access logs from cPanel. On a CDN, export logs from the dashboard or API, or use the provider's bot report.

Can I analyze crawler logs without command-line tools?

Yes. GoAccess and AWStats are open-source analyzers with visual reports. CDN dashboards also show bot traffic. The trade-off is less control over custom filters and, for some SaaS tools, a subscription cost.

Data sources: Google Search Central IP ranges documentation, OpenAI crawler IP documentation, Cloudflare verified bots list, and Apache and Nginx access log documentation.