Crawler reports

A dated, first-hand record of what crawlers did on this site, taken from our own access log and refreshed every day. One site's observations only: no benchmarks, no client data and no projections.

Latest snapshot

  • Refreshed: 2026-11-21 08:05 EST (US Eastern)
  • Total requests, trailing 24 hours: 7767
  • Recognised crawler requests: 201
  • Distinct visiting addresses: 692
  • Requests to known scan paths: 603 (blocked at the edge, listed here only to show the noise a public site attracts)
Recognised crawler share in the latest snapshotHorizontal bars showing the percentage of recognised crawler requests per named engine.Applebot23.4%Yandex20.4%Google18.9%Amazonbot15.4%Bing13.9%OAI-SearchBot3.5%GPTBot2.5%PerplexityBot1.0%
CrawlerRequestsShare
Applebot4723.4%
Yandex4120.4%
Google3818.9%
Amazonbot3115.4%
Bing2813.9%
OAI-SearchBot73.5%
GPTBot52.5%
PerplexityBot21.0%
Bytespider21.0%

What changed since the last refresh

  • Total requests in the trailing 24 hours: -8126 since the previous refresh.
  • Recognised crawler requests: -1059 since the previous refresh.
  • Distinct visiting addresses: +7 since the previous refresh.
  • The most active named engine is Applebot.
  • Most automation still arrives under a generic or unidentified agent. That is normal for a young site and is why the named figures matter more than the total.

Most requested paths

  • /data/live.json — 658 requests
  • / — 337 requests
  • /data/links.json — 32 requests
  • /feed.xml — 21 requests
  • /robots.txt — 12 requests

Where requests came from

CN 3131, BE 1591, SE 608, DE 489, US 383

What this does not mean

  • A higher crawl rate is not a citation, a ranking or demand. It is a precondition at best.
  • User agents can be spoofed, so unidentified traffic is reported as unverified rather than counted as a named engine.
  • These figures describe one site. They are not a benchmark and should never be presented as one.

How to check this page

  • The raw file behind every number is /data/live.json, updated every five minutes.
  • Visitor IP addresses are masked before publication. No third-party analytics are used, so there is no second set of numbers.
  • If this page and the raw file ever disagree, the raw file is right.

Period reports

Aggregated windows (week, month, quarter, year), built from the daily records so a single odd day cannot dominate. Each page carries its own data file.

Daily records

One dated record a day, written from the same access log as this page. These are the raw material the period reports above aggregate.

  • Live view digests — every daily record, newest first, each with the figures behind it.

Regenerated daily from the live data file. Last written 2026-11-21 08:10 EST.