Before installing this instrumentation anywhere else, we ran it here. Everything in this case is publicly checkable at /live/, /data/live.json and /reports/, which is the point: a case study backed by a live file cannot quietly be embellished later.
We tested every claim in this series on our own single-server site and published the numbers.
What the instrumentation showed
| Measure | Value | Reading |
|---|---|---|
| Requests, 24 hours | 65,084 | denominator, all traffic |
| Recognised crawler requests | 26,178 | 40% of the window |
| Distinct named engines | 10 | led by one AI crawler |
| Unrecognised automation | 31,000+ | reported separately, never merged |
| Known scan paths | 29,000+ | blocked at the edge, excluded |
Three findings were worth acting on: the largest single share of traffic was not search or AI at all but unrecognised automation; one AI crawler accounted for most of the named activity; and a meaningful volume of requests went to paths that only a scanner would ask for.
What changed because of it
- The crawler policy became explicit rather than implied: one Content-Signal line, one named crawler group per purpose.
- Retired URL patterns stopped being a silent question. Missing paths are now a published figure instead of a surprise during an audit.
- Scan traffic was separated from crawler traffic at the edge, so the crawler numbers stopped flattering themselves.
- Publishing the numbers in public turned out to be the strongest argument for the method: it forces every claim to be traceable.
What was unflattering
- Five days of record is far too short to call anything a trend, and we say so on every report.
- During this exercise we found our own robots.txt and llms.txt contradicting each other on AI training. It was published, found by an outside review, and fixed at the source rather than quietly.
- Most traffic is unverifiable by design. There is no honest way to turn that into a tidy chart.
Why this is a case study and not a showcase
There is no client logo here, no percentage improvement and no timeline graphic. There is a site, an access log, a published method and the figures those produce. If you want the same reading of your own site, the AI Crawler Log Analyzer is where we would start.
All figures from this site's own nginx access log, 26 September 2026, published continuously at /live/. One site only: no benchmarks and no projections.Related reading: Five Layers of Scanner Defence, Tested on Our Own Site · Layer 5: Closing the Loop in Public
More on this site: live signals · dated reports · how to verify us