https://lakora.net/article/ Practical explanations of how search engines and AI answer engines find, read and cite websites. en Sun, 27 Sep 2026 00:00:00 +0000 https://lakora.net/article/freshness-signals-what-ai-crawlers-notice.html https://lakora.net/article/freshness-signals-what-ai-crawlers-notice.html Feeds, sitemap lastmod and updated timestamps do three different jobs. Here is how we keep ours honest, and which parts we can and cannot measure. Sun, 27 Sep 2026 00:00:00 +0000 https://lakora.net/article/what-happened-when-we-blocked-scanner-traffic.html https://lakora.net/article/what-happened-when-we-blocked-scanner-traffic.html We added explicit 403 rules for scanner paths on our own site. Here is what moved afterwards, what did not, and the definition trap we found in our own dashboard. Sun, 27 Sep 2026 00:00:00 +0000 https://lakora.net/article/how-we-audit-a-site-the-checklist-we-use.html https://lakora.net/article/how-we-audit-a-site-the-checklist-we-use.html The exact checks we run before quoting work on a site: crawler access, structured data, freshness signals, sitemap integrity, and the parts no audit can prove. Sun, 27 Sep 2026 00:00:00 +0000 https://lakora.net/article/why-one-sites-crawl-numbers-are-not-a-benchmark.html https://lakora.net/article/why-one-sites-crawl-numbers-are-not-a-benchmark.html Crawl rates depend on age, authority and how often pages change. We publish our own numbers as a dated record, never as an industry average, and here is what that distinction protects you from. Sun, 27 Sep 2026 00:00:00 +0000 https://lakora.net/article/what-a-content-signal-line-does.html https://lakora.net/article/what-a-content-signal-line-does.html One line in robots.txt can state your search, AI input and AI training preferences. Here is what it actually changes, what it does not, and how to stop it contradicting your llms.txt. Sun, 27 Sep 2026 00:00:00 +0000 https://lakora.net/article/how-to-read-your-own-server-log-for-ai-crawlers.html https://lakora.net/article/how-to-read-your-own-server-log-for-ai-crawlers.html Your server log already records every AI crawler that visits. Here is what to look for, what the numbers cannot tell you, and how to separate real crawlers from ordinary automation. Sun, 27 Sep 2026 00:00:00 +0000 https://lakora.net/reports/live/2026-10-22/ https://lakora.net/reports/live/2026-10-22/ 1,758 of the 2,184 requests in the current window carry a user agent that matches no recognised search engine or AI crawler. Thu, 22 Oct 2026 12:00:17 +0000 https://lakora.net/article/how-ai-answer-engines-choose-which-pages-to-cite.html https://lakora.net/article/how-ai-answer-engines-choose-which-pages-to-cite.html AI answer engines retrieve and rank passages to select citations. Learn how the process works, what makes content citable, and how to audit your own pages. Wed, 23 Sep 2026 00:00:00 +0000 https://lakora.net/article/how-googlebot-decides-what-to-crawl-first.html https://lakora.net/article/how-googlebot-decides-what-to-crawl-first.html A practical breakdown of how Googlebot decides what to crawl, based on public Google guidance, plus a pre-launch checklist and monitoring tools. Wed, 23 Sep 2026 00:00:00 +0000 https://lakora.net/reports/live/2026-10-21/ https://lakora.net/reports/live/2026-10-21/ OAI-SearchBot made 541 requests in the current window, against a recent daily average of 334 for this site. The comparison is against this site only, not against other sites. Wed, 21 Oct 2026 12:00:32 +0000 https://lakora.net/reports/live/2026-10-20/ https://lakora.net/reports/live/2026-10-20/ The most requested missing path in the current window is /app with 0 requests. Retired or never-published URLs stay in circulation long after they stop existing. Tue, 20 Oct 2026 12:00:31 +0000 https://lakora.net/reports/live/2026-10-19/ https://lakora.net/reports/live/2026-10-19/ / took 660 crawler requests in the current window. That is 78% above its recent daily average. Mon, 19 Oct 2026 12:00:30 +0000 https://lakora.net/reports/live/2026-10-18/ https://lakora.net/reports/live/2026-10-18/ Of the 11,530 requests in the current 24-hour window, 3,169 came from recognised crawlers and AI agents (27.5% of all traffic). That share is 14% below the average of the previous 26 days. Sun, 18 Oct 2026 12:00:15 +0000 https://lakora.net/reports/live/2026-10-17/ https://lakora.net/reports/live/2026-10-17/ Google is the most active named crawler in the current 24-hour window at 32 requests, 96% below its average over the previous 25 recorded days. Sat, 17 Oct 2026 12:00:22 +0000 https://lakora.net/reports/live/2026-10-16/ https://lakora.net/reports/live/2026-10-16/ 1,110 requests in the current window went to paths only a scanner would ask for, and were blocked at the edge. They are published separately so they never inflate the crawler figures. Fri, 16 Oct 2026 12:00:15 +0000 https://lakora.net/reports/live/2026-10-15/ https://lakora.net/reports/live/2026-10-15/ ChatGPT-User shows up in the current 24-hour window with 164 requests and had not appeared in any previous daily aggregate for this site. Thu, 15 Oct 2026 12:00:22 +0000 https://lakora.net/reports/live/2026-10-14/ https://lakora.net/reports/live/2026-10-14/ The last completed day recorded 10,897 requests, against a 21,603-request average across the earlier days in this record. That is 50% lower. Wed, 14 Oct 2026 12:00:25 +0000 https://lakora.net/reports/live/2026-10-13/ https://lakora.net/reports/live/2026-10-13/ Requests resolve to 8 countries in the current window, led by BR with 9,705 requests. Tue, 13 Oct 2026 12:00:33 +0000 https://lakora.net/reports/live/2026-10-12/ https://lakora.net/reports/live/2026-10-12/ 4,452 of the 5,719 requests in the current window carry a user agent that matches no recognised search engine or AI crawler. Mon, 12 Oct 2026 12:00:26 +0000 https://lakora.net/reports/live/2026-10-11/ https://lakora.net/reports/live/2026-10-11/ meta-externalagent made 1,037 requests in the current window, against a recent daily average of 681 for this site. The comparison is against this site only, not against other sites. Sun, 11 Oct 2026 12:00:30 +0000 https://lakora.net/reports/live/2026-10-10/ https://lakora.net/reports/live/2026-10-10/ The most-crawled path in the current window is /article/ with 104 requests, against a recent daily average of 52 for the same path on this site. Sat, 10 Oct 2026 12:00:16 +0000 https://lakora.net/reports/live/2026-10-09/ https://lakora.net/reports/live/2026-10-09/ Of the 11,411 requests in the current 24-hour window, 4,042 came from recognised crawlers and AI agents (35.4% of all traffic). That share is 12% above the average of the previous 17 days. Fri, 09 Oct 2026 12:00:29 +0000 https://lakora.net/reports/live/2026-10-08/ https://lakora.net/reports/live/2026-10-08/ 5,587 requests in the current window came from 249 distinct addresses, an average of 22.4 requests each. Shared address ranges lower this figure; a single busy crawler raises it. Thu, 08 Oct 2026 12:00:21 +0000 https://lakora.net/reports/live/2026-10-07/ https://lakora.net/reports/live/2026-10-07/ 2,240 requests in the current window went to paths only a scanner would ask for, and were blocked at the edge. They are published separately so they never inflate the crawler figures. Wed, 07 Oct 2026 12:00:23 +0000 https://lakora.net/reports/live/2026-10-06/ https://lakora.net/reports/live/2026-10-06/ ChatGPT-User shows up in the current 24-hour window with 154 requests and had not appeared in any previous daily aggregate for this site. Tue, 06 Oct 2026 12:00:19 +0000 https://lakora.net/reports/live/2026-10-05/ https://lakora.net/reports/live/2026-10-05/ ClaudeBot is the most active named crawler in the current 24-hour window at 286 requests, 86% below its average over the previous 13 recorded days. Mon, 05 Oct 2026 12:00:13 +0000 https://lakora.net/reports/live/2026-10-04/ https://lakora.net/reports/live/2026-10-04/ Requests resolve to 8 countries in the current window, led by HU with 465 requests. Sun, 04 Oct 2026 12:00:08 +0000 https://lakora.net/article/five-layers-of-scanner-defence-tested-on-our-own-site.html https://lakora.net/article/five-layers-of-scanner-defence-tested-on-our-own-site.html Edge blocking, server hardening, behaviour analysis, honeypots and public reporting - the series in one model, with the one place where our practice diverges from the usual advice for each layer. Sun, 04 Oct 2026 00:00:00 +0000 https://lakora.net/reports/live/2026-10-03/ https://lakora.net/reports/live/2026-10-03/ 1,483 of the 1,805 requests in the current window carry a user agent that matches no recognised search engine or AI crawler. Sat, 03 Oct 2026 12:00:08 +0000 https://lakora.net/article/layer-5-closing-the-loop-in-public.html https://lakora.net/article/layer-5-closing-the-loop-in-public.html The fifth layer is not a control at all - it is the reporting. Publishing refusal counts next to their definitions is what turns four layers of defence into something a reader can check. Sat, 03 Oct 2026 00:00:00 +0000 https://lakora.net/reports/live/2026-10-02/ https://lakora.net/reports/live/2026-10-02/ The last completed day recorded 13,802 requests, against a 35,789-request average across the earlier days in this record. That is 61% lower. Fri, 02 Oct 2026 12:00:10 +0000 https://lakora.net/article/layer-4-honeypots-as-early-warning.html https://lakora.net/article/layer-4-honeypots-as-early-warning.html A decoy path nobody legitimate would ever request is a high-signal tripwire. It is not a fingerprinting laboratory - here is what we actually learn, and the two rules that keep a honeypot from becoming a liability. Fri, 02 Oct 2026 00:00:00 +0000 https://lakora.net/reports/live/2026-10-01/ https://lakora.net/reports/live/2026-10-01/ /data/live.json took 340 crawler requests in the current window. That is 72% below its recent daily average. Thu, 01 Oct 2026 12:00:11 +0000 https://lakora.net/article/layer-3-behaviour-analysis-without-hurting-crawlers.html https://lakora.net/article/layer-3-behaviour-analysis-without-hurting-crawlers.html Rate limiting and automatic bans sound like precision tools. On a site that depends on being crawled, the precision is mostly in the whitelist - and that is exactly where they get dangerous. Thu, 01 Oct 2026 00:00:00 +0000 https://lakora.net/reports/live/2026-09-30/ https://lakora.net/reports/live/2026-09-30/ Of the 5,088 requests in the current 24-hour window, 957 came from recognised crawlers and AI agents (18.8% of all traffic). That share is 48% below the average of the previous 8 days. Wed, 30 Sep 2026 12:00:13 +0000 https://lakora.net/article/layer-2-server-hardening-and-the-403-versus-444-question.html https://lakora.net/article/layer-2-server-hardening-and-the-403-versus-444-question.html Hardening is the layer you control completely - and the one where the textbook advice (drop the connection with 444) collides with the value of a published, verifiable refusal. Wed, 30 Sep 2026 00:00:00 +0000 https://lakora.net/reports/live/2026-09-29/ https://lakora.net/reports/live/2026-09-29/ The most-crawled path in the current window is /article/ with 187 requests, against a recent daily average of 56 for the same path on this site. Tue, 29 Sep 2026 12:00:18 +0000 https://lakora.net/article/layer-1-blocking-at-the-edge-and-why-we-do-not-challenge.html https://lakora.net/article/layer-1-blocking-at-the-edge-and-why-we-do-not-challenge.html The edge is the only layer that changes the volume, because the volume is measured at the origin. We block probe paths there and deliberately do not use a challenge - here is the evidence behind both choices. Tue, 29 Sep 2026 00:00:00 +0000 https://lakora.net/reports/live/2026-09-28/ https://lakora.net/reports/live/2026-09-28/ 4,191 requests in the current window went to paths only a scanner would ask for, and were blocked at the edge. They are published separately so they never inflate the crawler figures. Mon, 28 Sep 2026 12:00:23 +0000 https://lakora.net/article/why-we-moved-scanner-blocking-to-the-edge.html https://lakora.net/article/why-we-moved-scanner-blocking-to-the-edge.html Refusing scanners at the origin kept our published count high, because the count measures what reaches us. Moving the decision to the edge is the only layer that changes that - and the plan we are on shaped how the rule had to be written. Mon, 28 Sep 2026 00:00:00 +0000