The third layer looks at patterns instead of paths: how often, from where, with which user agent, in what order. It is the layer with the best reputation and the worst failure mode.
Behaviour beats a user-agent string, because one IP range carries both crawlers and scanners.
What this layer is
- Rate limits: a visitor, an address or an agent may only do so much per minute before requests are delayed or dropped.
- Behavioural bans: repeated hits on sensitive paths trigger a temporary block.
- Slow and distributed scanning: many addresses, low rates, patient coverage - visible only across a long window.
What we run
We rate-limit our one state-changing endpoint (the enquiry form) at six requests per minute per address with a small burst, which is generous for humans and useless for automation. We do not auto-ban scanners, and that is a deliberate choice rather than an omission.
Where we disagree with the textbook
Automatic IP bans assume that a bad IP is a bad IP. Our own classification says otherwise: every top source address in our logs carries both crawler and scanner traffic, because both run from the same cloud ranges. A ban list built on volume would have removed the crawlers this site depends on. The usual mitigation is to whitelist verified crawler ranges — which works, but the whitelist is now load-bearing infrastructure, and user agents alone are trivially forged.
Slow, distributed scans are the harder problem. From a single origin log you can see that a pattern exists across many addresses; you cannot honestly attribute intent, and you cannot cleanly separate a patient scanner from a slow legitimate client. We publish the trend instead of pretending to a verdict.
How to verify
- Rate limiting is visible in our lead endpoint behaviour; the service logs every delivery attempt and window.
- The classification method is described in our own reporting; the same numbers appear in /data/live.json and the frozen record under /reports/.
- Request volumes per engine are published as we count them, unverified automation included.
Is rate limiting enough to stop scanning?
No. It bounds abuse of endpoints worth abusing; path refusal is what removes scanner noise, because scanners do not repeat the same path often.
Should I ban scanning IPs automatically?
Only if you can whitelist real crawlers and accept the maintenance. If your crawler and scanner traffic share ranges - as ours does - volume is not a safe signal.
Can I detect a distributed scan?
You can observe that a pattern spans many addresses. Attribution is a different claim, and we do not make it.
Related reading: Five Layers of Scanner Defence, Tested on Our Own Site · Layer 5: Closing the Loop in Public
More on this site: live signals · dated reports · how to verify us