A public site attracts two kinds of traffic that have nothing to do with readers: AI and search crawlers reading pages, and scanners looking for something to break into. We block the second group explicitly, and this week we widened the rules. Here are the numbers before and after, from our own log.
Blocking probe traffic at the edge cut scanner requests while crawler access stayed unchanged.
What we changed
- Added the scanner paths our own log showed slipping through: versioned API probes, build artefacts, framework endpoints and credential files.
- Refused one class of path by name (anything containing a GraphQL endpoint marker) rather than by directory.
- Kept the response a plain
403with a tiny body instead of closing the connection: we would rather publish what we block than hide it.
What moved
| Measure | Before | After | Why |
|---|---|---|---|
| Scanner paths returning 404 | falling through to the 404 handler | now refused with 403 | the rule set covers the paths the log actually showed |
| Requests refused with 403 | ~37,000 / 24h | ~38,000 / 24h | the same traffic, answered more precisely |
| 404 responses | ~29,000 / 24h | ~29,300 / 24h | unrelated to the change; the 404 bucket is status-based |
What did not move, and why that matters
The headline number of blocked traffic did not fall. That is the honest result: our counter counts requests to known scanner paths regardless of status code, and the requests still arrive at the server before being refused. Blocking at the origin makes the answer cheaper and the log cleaner; it does not stop the traffic. Only an edge rule at the CDN layer can do that, and we say so instead of implying a win we did not get.
The definition trap we found in our own dashboard
While checking the numbers we found two panels that looked like the same metric and were not: one counted scanner paths by pattern (31,961 in the window), the other counted HTTP 404 responses (28,399). Both were labelled correctly, but they invite the wrong comparison, because a request that is refused with 403 leaves the 404 bucket while staying in the scan bucket. Writing the definitions next to the numbers is the fix; the definitions now ship inside the data file itself.
What we take from this
- Refuse early, refuse precisely, and keep the response small.
- Publish the definition of every number you publish, or the numbers will be compared incorrectly by someone else.
- Expect the blocked-traffic counter to keep rising: it measures noise attracted, not noise defeated.
Did blocking scanner paths reduce your bandwidth?
Slightly, because refused requests return a short body instead of a full 404 page. The traffic itself still arrives; the saving is in the answer, not the arrival.
Should I block scanners in robots.txt too?
robots.txt is a request, and scanners are precisely the clients that ignore it. Use it to state your policy for well-behaved crawlers, and handle the rest at the edge.
Do these rules affect search or AI crawlers?
No. Search engines and AI retrieval agents are explicitly allowed; the rules match scanner paths, not user agents.
Related reading: Five Layers of Scanner Defence, Tested on Our Own Site · Layer 5: Closing the Loop in Public
More on this site: live signals · dated reports · how to verify us