Crawler reports
A dated, first-hand record of what crawlers did on this site, taken from our own access log and refreshed every day. One site's observations only: no benchmarks, no client data and no projections.
Latest snapshot
- Refreshed: 2026-11-21 08:05 EST (US Eastern)
- Total requests, trailing 24 hours: 7767
- Recognised crawler requests: 201
- Distinct visiting addresses: 692
- Requests to known scan paths: 603 (blocked at the edge, listed here only to show the noise a public site attracts)
| Crawler | Requests | Share |
|---|---|---|
| Applebot | 47 | 23.4% |
| Yandex | 41 | 20.4% |
| 38 | 18.9% | |
| Amazonbot | 31 | 15.4% |
| Bing | 28 | 13.9% |
| OAI-SearchBot | 7 | 3.5% |
| GPTBot | 5 | 2.5% |
| PerplexityBot | 2 | 1.0% |
| Bytespider | 2 | 1.0% |
What changed since the last refresh
- Total requests in the trailing 24 hours: -8126 since the previous refresh.
- Recognised crawler requests: -1059 since the previous refresh.
- Distinct visiting addresses: +7 since the previous refresh.
- The most active named engine is Applebot.
- Most automation still arrives under a generic or unidentified agent. That is normal for a young site and is why the named figures matter more than the total.
Most requested paths
/data/live.json— 658 requests/— 337 requests/data/links.json— 32 requests/feed.xml— 21 requests/robots.txt— 12 requests
Where requests came from
CN 3131, BE 1591, SE 608, DE 489, US 383
What this does not mean
- A higher crawl rate is not a citation, a ranking or demand. It is a precondition at best.
- User agents can be spoofed, so unidentified traffic is reported as unverified rather than counted as a named engine.
- These figures describe one site. They are not a benchmark and should never be presented as one.
How to check this page
- The raw file behind every number is /data/live.json, updated every five minutes.
- Visitor IP addresses are masked before publication. No third-party analytics are used, so there is no second set of numbers.
- If this page and the raw file ever disagree, the raw file is right.
Period reports
Aggregated windows (week, month, quarter, year), built from the daily records so a single odd day cannot dominate. Each page carries its own data file.
- 2026, week 47 (2026-11-16 to 2026-11-22)
- 2026, week 46 (2026-11-09 to 2026-11-15)
- 2026, week 45 (2026-11-02 to 2026-11-08)
- 2026, week 44 (2026-10-26 to 2026-11-01)
- 2026, week 43 (2026-10-19 to 2026-10-25)
- 2026, week 42 (2026-10-12 to 2026-10-18)
- 2026, week 41 (2026-10-05 to 2026-10-11)
- 2026, week 40 (2026-09-28 to 2026-10-04)
- 2026, week 39 (2026-09-21 to 2026-09-27) — partial period (data begins 2026-09-21)
- October 2026
Daily records
One dated record a day, written from the same access log as this page. These are the raw material the period reports above aggregate.
- Live view digests — every daily record, newest first, each with the figures behind it.
Regenerated daily from the live data file. Last written 2026-11-21 08:10 EST.
