Refusing probe traffic is the part everyone agrees on. Where you refuse decides whether your own numbers move at all. This is part one of a five-part series on scanner defence, written from our own logs rather than from theory.
Refuse probe traffic at the edge, but never challenge a client that cannot run JavaScript.
What this layer is
A CDN or WAF evaluates the request before it reaches your server, matching path fingerprints, user agents or reputation signals, and answers with a block or a challenge. Because the request never arrives, it never reaches your access log.
What our own numbers showed
In our own window: 71,191 requests, of which 40,263 were refused and 33,133 matched known scan paths; the top five probe paths were /v1/graphql (857), /api/graphql (845), /graphql (831), /api/config (814) and /actuator/loggers (533).
Refusing those requests at the origin did not reduce the count — by definition, because the count reads the origin's log. Moving the same refusal to the edge changed which layer answers: our probe paths now return a provider block page, and the origin-side counters are the ones that should fall.
What we run
- Path families: credential files, framework endpoints and the scanner patterns our log actually showed.
- Extensions we never serve: .php, .py, .yml, .bak, .sql.
- Scraper user agents we previously refused at the origin, now refused one layer earlier.
Where we disagree with the textbook
The usual advice is to answer with a JavaScript challenge instead of a hard block. For a site whose purpose is being read and cited by AI answer engines, that is the wrong tool: retrieval crawlers do not run JavaScript, so a challenge is a block for exactly the clients we want. We tested this — requests identifying as GPTBot, Googlebot and ClaudeBot are served normally on this site. A challenge belongs on human-facing abuse (login attempts, form spam), not on probe paths.
The second disagreement is about reputation. Blocking by IP or ASN sounds precise; we classified our own traffic first and every top source address turned out to be a cloud range carrying both crawler and scanner traffic. Blocking those ranges would have removed readers along with noise. Path fingerprints do not have that problem.
How to verify
- Edge vs Origin Check tells you, per probe path, whether a provider block page, the origin or nothing answered.
- /data/live.json publishes the refusal counts together with their definitions.
- /data/health.json shows whether our collector is current.
Does edge blocking stop the scanners?
It stops them reaching your server, which is the part you control. The attempts continue; they simply never arrive.
Why not challenge every suspicious request?
Because challenged clients that do not run JavaScript - including AI retrieval crawlers - are treated as failures. On a site that earns its traffic by being crawlable, that is self-harm.
Is a provider block the same as a 403?
No. A 403 comes from your server and appears in your log; a provider block answers before your server sees the request.
Related reading: Five Layers of Scanner Defence, Tested on Our Own Site · Layer 5: Closing the Loop in Public
More on this site: live signals · dated reports · how to verify us