Case study: making crawler traffic visible
A manufacturer relaunched its site with no way to tell which search engines and AI agents were reading it. We instrumented the server, built a dashboard, and published the numbers. This case is a real site with a live dashboard, not a composite example.
What could not be answered before
- Which engines crawl the site, how often, and which pages they take.
- How much of the raw request count is scanners and probes rather than readers.
- Whether AI assistants fetch the pages we want cited, or only competitors.
What we built
- Server-side log parsing only: no analytics product, no cookies, no client-side script.
- A nine-panel dashboard: hourly volume, top paths, engine mix, 7-day trend and share, per-day series, recent hits, search referrers and hourly engine activity.
- Edge rules that block known scanner fingerprints, so the figures describe readers rather than noise.
- An English interface behind a token, so international teams read the same numbers as the operators.
What it shows
Figures read from the live dashboard on 23 September 2026:
- 824 recognised crawler visits in the previous 24 hours.
- Seven-day totals by engine: Amazonbot 7,647 (435 addresses), Other 1,822, Bytespider 980 (508 addresses), Bing 694, Google 342, ClaudeBot 313.
- Address counts per engine differ by two orders of magnitude: Google and ClaudeBot arrive from a handful of addresses, Bytespider from hundreds. That tells you which engines come from fixed infrastructure and which do not.
Why it matters
What gets crawled determines what can be cited. Without this view it is easy to rewrite pages that no engine reads and to leave the pages engines do read half-specified.
See it for yourself
The dashboard is live and token-gated — access on request. Our own equivalent is public at /live/, and the measurement method is described at /reports/. This instrumentation is how every engagement starts.