AI crawler report — 2026, week 39 (2026-09-21 to 2026-09-27)

4 day(s) of data in this period, read directly from this site's own access log. This is a partial period: data begins 2026-09-22, when site instrumentation started.

Summary

  • 122,516 total requests, of which 45,432 were recognised crawler requests (37.1%).
  • 7 distinct crawlers identified. Largest: ClaudeBot (9,255 requests).

Crawlers by engine

ClaudeBot9,255PerplexityBot6,275Google3,791GPTBot3,584meta-externalagent2,895OAI-SearchBot860CCBot272
Requests by engine, 2026, week 39 (2026-09-21 to 2026-09-27). AI answer-engine and AI-training crawlers are highlighted; search engines are grey.
CrawlerRequestsShare of crawler traffic
ClaudeBot9,25520.4%
PerplexityBot6,27513.8%
Google3,7918.3%
GPTBot3,5847.9%
meta-externalagent2,8956.4%
OAI-SearchBot8601.9%
CCBot2720.6%

Most-crawled pages

PathRequests
/data/live.json5,720
/2,389
/article/how-googlebot-decides-what-to-crawl-first.html440
/disclaimer/384
/tools/ai-policy-files/370
/manifest.json332
/article/how-ai-answer-engines-choose-which-pages-to-cite.html316
/live/208
/tools/124
/llms.txt52
/reports/48
/case/shimigao/43

Method, and what this is not

  • Source: our own nginx access log, aggregated per day, then summed for this period. No third-party analytics, no cookies, no client script.
  • Crawlers are identified by user agent string. This can over-count: some automated traffic claims a crawler identity. We treat these numbers as indicative, not audited.
  • One site only. This is not a benchmark, index, or projection. It is a dated, first-hand record — useful precisely because it is small, local and verifiable.
  • Data file for this period: /reports/data/2026-w39.json

See also: live crawler view · all reports