We publish what arrives at this site, so we can also see what our anti-scanner work does and does not do. Refusing probe traffic at the origin - which we have done since the beginning - turned out to have a blind spot: the published count kept rising even though we were refusing everything.
The origin should refuse what reaches it; the edge should stop the crowd before it arrives.
Why the origin was not enough
Our counters read the server's own access log. A request we refuse with 403 is still a request that arrived, so it still appears in the figures. Refusing politely at the origin makes the answer cheap and the log honest; it does not change the volume. The only way to change the volume is to decide earlier - before the request reaches this server.
What the edge rule does
- Probe families by path. Credential files (
/.env,/.git,/.aws), framework endpoints (/actuator,/graphql,/instance/config,/feast/) and the scanner patterns our own log actually showed. - Script and config extensions we never serve:
.php,.py,.yml,.bak,.sql. - Scraper user agents we already refused at the origin, now refused one layer earlier.
What we deliberately did not do
- No blocking by country or by IP range. We classified our own traffic first and found that every top source address is a cloud provider address carrying both crawler and scanner traffic. Blocking those ranges would have cut the crawlers we depend on.
- No blanket bot challenges. AI retrieval crawlers are how this site gets read and cited. A challenge that stops them defeats the point of the site.
- No attempt to reach zero scan traffic. Noise is a constant on a public address. The goal is cheap refusal plus visibility, not a fantasy of silence.
The plan detail we did not expect
The rule language offers a regex operator (matches) on higher plans only; our plan refused it outright with not entitled. The same coverage is expressible with contains, which every plan has, so the deployed rule uses that. Worth knowing before writing anything elaborate: on a small site, string containment is usually enough.
What we expect the numbers to do
Refused earlier, the requests should stop reaching the origin, so the origin-side counters - the ones we publish - should fall instead of staying high. We will publish what actually happens in the dated record rather than predicting it here, and the edge keeps its own count that we cannot see from inside.
How to check any of this yourself
- Fetch /data/live.json and read
security: how many requests were refused, and against which definitions. - Read /data/health.json: whether our collector is current, and how old its last numbers are.
- Compare /live/ with /reports/: live figures against the frozen dated record.
- Run our own Edge vs Origin Check on any domain: it reports, per probe path, whether a provider block page, the origin, or nothing answered - and whether a CDN is in front.
Does edge blocking stop the scanners?
It stops them reaching us, which is what we control. It does not stop them trying; the attempts simply never arrive.
Why not block the noisiest addresses?
Because we checked what they carry. The top addresses in our log carry both crawler and scanner traffic, so blocking them would remove readers along with noise.
Will your published numbers fall now?
The origin-side counters should, because refused-before-arrival requests are no longer in the log. The dated record will show whether that happened.
Related reading: Five Layers of Scanner Defence, Tested on Our Own Site · Layer 5: Closing the Loop in Public
More on this site: live signals · dated reports · how to verify us