We publish our crawler numbers in public, and we are careful about one sentence every time: these figures describe one small site. That is not modesty, it is method. Here is what goes wrong when a single site's numbers are read as an industry average.
One site's crawl volume describes that site only and cannot be compared with another.
What our own numbers look like
| Measure | Our window, 26 September 2026 |
|---|---|
| Requests, 24 hours | 65,084 |
| Recognised crawler requests | 26,178 |
| Distinct named engines | 10 |
| Days in the record | 5 |
| Site age at recording | about two weeks |
Why they cannot be generalised
- Crawl rate scales with links and authority. A two-week-old site is simply not crawled at the rate of an established one.
- It depends on how often pages change. A site that publishes daily is recrawled more often than a static brochure, for reasons that have nothing to do with AI.
- It depends on what the site is about and who links to it. Technologies and B2B services attract different crawler mixes than retail or media.
- Five days of data cannot establish a season, a trend or a baseline. It can only establish what happened in those five days.
What you can legitimately do with them
- Compare a site against itself, over time. That comparison is valid and useful.
- Check whether a specific crawler reaches you at all, and whether your robots.txt is doing what you think.
- Set expectations about the shape of crawler traffic on a young site, without attaching invented percentages to it.
- Verify a number instead of trusting it: every figure we publish has a raw file behind it.
How to check us
The live numbers are at /live/, the raw aggregate is /data/live.json, and the dated records live at /reports/. If a figure on this site disagrees with the raw file, the raw file is right.
Source: all figures in this note come from this site's own nginx access log, aggregated every five minutes. One site's observations only: no benchmarks, no projections.Then why publish numbers at all?
Because a verifiable record is more useful than a confident claim, and because most crawler data in circulation is unauditable. Ours is small, dated and checkable.
Can I use your figures in a comparison?
Not as a market baseline. You may quote them as one dated first-hand record, with the date and the source attached.
How long until the record is meaningful?
For a site's own trend, a few weeks. For any claim about a market, never, from one site.
Related reading: Five Layers of Scanner Defence, Tested on Our Own Site · Layer 5: Closing the Loop in Public
More on this site: live signals · dated reports · how to verify us