AI crawler report — 2026, week 44 (2026-10-26 to 2026-11-01)
7 day(s) of data in this period, read directly from this site's own access log. Complete period.
Summary
- 86,567 total requests, of which 16,480 were recognised crawler requests (19.0%).
- 8 distinct crawlers identified. Largest: ClaudeBot (2,931 requests).
Crawlers by engine
| Crawler | Requests | Share of crawler traffic |
|---|---|---|
| ClaudeBot | 2,931 | 17.8% |
| 2,289 | 13.9% | |
| meta-externalagent | 1,974 | 12.0% |
| Bing | 1,367 | 8.3% |
| GPTBot | 1,054 | 6.4% |
| Amazonbot | 103 | 0.6% |
| OAI-SearchBot | 37 | 0.2% |
| Yandex | 14 | 0.1% |
Most-crawled pages
| Path | Requests |
|---|---|
| / | 1,713 |
| /sitemap.xml | 341 |
| /data/live.json | 339 |
| /article/ | 297 |
| /tools/ | 284 |
| /case/ | 281 |
| /tools/schema-generator/ | 275 |
| /disclaimer/ | 273 |
| /guide/dead-link-detection-and-repair.html | 214 |
| /guide/complete-website-internal-link-strategy.html | 210 |
| /guide/complete-website-redesign-and-domain-migration-process.html | 193 |
| /article/crawled-but-not-indexed-fix-small-site-pages.html | 112 |
Method, and what this is not
- Source: our own nginx access log, aggregated per day, then summed for this period. No third-party analytics, no cookies, no client script.
- Crawlers are identified by user agent string. This can over-count: some automated traffic claims a crawler identity. We treat these numbers as indicative, not audited.
- One site only. This is not a benchmark, index, or projection. It is a dated, first-hand record — useful precisely because it is small, local and verifiable.
- Data file for this period: /reports/data/2026-w44.json
See also: live crawler view · all reports