Crawler reports

A dated, first-hand record of what crawlers did on this site, taken from our own access log and refreshed every day. One site's observations only: no benchmarks, no client data and no projections.

Latest snapshot

  • Refreshed: 2026-09-22 19:20 EDT (US Eastern)
  • Total requests, trailing 24 hours: 14390
  • Recognised crawler requests: 3595
  • Distinct visiting addresses: 448
  • Requests to known scan paths: 4881 (blocked at the edge, listed here only to show the noise a public site attracts)
Recognised crawler share in the latest snapshotHorizontal bars showing the percentage of recognised crawler requests per named engine.ClaudeBot19.5%PerplexityBot13.5%Google11.2%GPTBot9.5%CCBot7.2%meta-externalagent7.1%Amazonbot6.5%Bytespider6.4%
CrawlerRequestsShare
ClaudeBot70219.5%
PerplexityBot48513.5%
Google40111.2%
GPTBot3429.5%
CCBot2587.2%
meta-externalagent2577.1%
Amazonbot2356.5%
Bytespider2306.4%
ChatGPT-User2196.1%
OAI-SearchBot2005.6%

What changed since the last refresh

  • Total requests in the trailing 24 hours: +39 since the previous refresh.
  • Recognised crawler requests: +1 since the previous refresh.
  • Distinct visiting addresses: no change since the previous refresh.
  • The most active named engine is ClaudeBot.
  • Most automation still arrives under a generic or unidentified agent. That is normal for a young site and is why the named figures matter more than the total.

Most requested paths

  • /data/live.json — 1965 requests
  • / — 609 requests
  • /live/ — 140 requests
  • /manifest.json — 75 requests
  • /tools/ — 72 requests

Where requests came from

US 5844, SG 3491, CN 1957, DE 694, NL 353

What this does not mean

  • A higher crawl rate is not a citation, a ranking or demand. It is a precondition at best.
  • User agents can be spoofed, so unidentified traffic is reported as unverified rather than counted as a named engine.
  • These figures describe one site. They are not a benchmark and should never be presented as one.

How to check this page

  • The raw file behind every number is /data/live.json, updated every five minutes.
  • Visitor IP addresses are masked before publication. No third-party analytics are used, so there is no second set of numbers.
  • If this page and the raw file ever disagree, the raw file is right.

Regenerated daily from the live data file. Last written 2026-09-22 19:20 EDT.