Site map
Every page on this site, grouped so a reader or a crawler can reach any of them in one step.
- Home
- Live signals from our own network
- Live view digests
- Verify this site yourself
- Articles
- Crawler reports
- Case studies
- How we work
- Case study: Case study: our own network under the same instrumentation
- Case study: Case study: making crawler traffic visible
- About Page Trust: Before and After Rewrites
- Entity Consistency: Names, Addresses and Services Must Agree
- Experience vs Expertise: What Small Sites Can Prove
- How to Read Server Logs to Find Which Crawlers Visit
- JSON-LD vs Microdata: What Structured Data Changes
- Robots.txt, Content-Signal, and llms.txt: Make Them Agree
- Robots.txt Tokens for AI Crawlers, Explained
- Robots.txt vs Noindex: When Each One Actually Works
- Sandbox vs Penalty: Telling New-Site Problems Apart
- The Only 5 Schema Types Most Small Sites Need
- Three Types of Content Updates (and Which Moves Citations)
- Training vs Retrieval Crawlers: What Most People Get Wrong
- What Actually Gets Cited by AI Engines
- Entity Consistency: Names, Addresses, Services Must Agree
- Google Sandbox or Penalty? How to Tell
- How to Read Server Logs for AI Crawler Visits
- Keyword Density Is Dead (and Was Never That Useful)
- Months 1-3 on a New Site: What to Do When Nothing Happens
- Robots.txt for AI Crawlers: What Each Token Means
- Three Names Problem: When Your Brand Has Too Many Variants
- Training vs Retrieval Crawlers: Why the Difference Matters
- What Actually Gets Cited by AI Engines: Myths vs Reality
- What Structured Data Changes in Search and AI Answers
- Why 'Crawled' Does Not Mean 'Indexed'
- Why Your FAQ Schema Isn't Showing in Search Results
- Crawled Is Not Indexed: Fix the Gap
- Google Sandbox vs Penalty: 3 Checks to Tell Them Apart
- How to Read Server Logs for Crawler Visits
- How to Track AI Citations Manually (No Tool Needed)
- Robots.txt, Content-Signal, llms.txt: Make Them Agree
- The 5 Schema Types Your Small Site Actually Needs
- Training vs Retrieval Crawlers: What Most Teams Get Wrong
- What Gets Cited by AI: Correcting the Myths
- What Structured Data Changes: Rich Results vs Entity Clarity
- Why FAQ Schema Isn't Showing in Search Results
- Why Updating One Statistic Beats Writing a New Article
- Crawled Not Indexed: Why It Happens and How to Fix It
- FAQ Schema Not Showing? The Real Trigger Condition
- Google Sandbox vs Penalty: What's Blocking Your New Site?
- Read Server Logs: Which Crawlers Hit Your Site
- Schema Types: The Only 5 a Small Site Really Needs
- Training vs Retrieval Crawlers: What Most Guides Get Wrong
- Update One Statistic vs Write New: What You Can Measure
- 5 Schema Types Most Small Sites Need
- Brand Name Variants: Myths vs Search Engine Evidence
- How to Read Server Logs to See Which Crawlers Visit
- Robots.txt Tokens for AI Crawlers: What They Do
- Google Sandbox vs Penalty: A Symptom Checklist
- How to Track AI Citations Manually: Anatomy of One Check
- Read Server Logs for Crawlers: Two Approaches
- Robots.txt for AI Crawlers: Token Guide
- The Three Names Problem: Brand Variant Myths vs Evidence
- Why Crawled Does Not Mean Indexed: The Pipeline
- Experience vs Expertise: What Small B2B Sites Can Prove
- How to Track AI Citations Manually: Monthly Routine
- Crawled but Not Indexed: Fix Small Site Pages
- Read Server Logs to Find Crawlers: Two Approaches
- Robots.txt Tokens for AI Crawlers: A Practical Walkthrough
- AI Crawler Types: Training vs Retrieval (Common Myths)
- Google Sandbox: What It Is and What It Isn't
- How to Read Server Logs for Crawler Visits: Two Methods
- Robots.txt Tokens for AI Crawlers: What Each Means
- What Structured Data Actually Changes (and What It Doesn't)
- 5 Schema Types Most Small Sites Actually Need
- Google Sandbox: New Site, Penalty or Bug?
- Structured Data: What It Changes vs What It Doesn't
- Training vs Retrieval Crawlers: What People Get Wrong
- Google Sandbox vs Penalty: How to Tell Them Apart
- Read Server Logs for AI Crawler Hits: Two Ways
- Robots.txt for AI Crawlers: Token by Token
- Entity Consistency: Names, Addresses, Services
- Robots.txt AI Crawlers: What Each Token Means
- Structured Data: What It Really Changes
- Crawled vs Indexed: What Practitioners Get Wrong
- Structured Data: What It Changes in Search and AI Answers
- Training Crawlers vs Retrieval Crawlers: A Practical Guide
- Only 5 Schema Types Your Small Site Needs
- Training vs Retrieval Crawlers: What Site Owners Get Wrong
- What Structured Data Changes: A B2B Decision Guide
- Five Layers of Scanner Defence, Tested on Our Own Site
- Layer 5: Closing the Loop in Public
- Layer 4: Honeypots as Early Warning (and Their Limits)
- Layer 3: Behaviour Analysis Without Hurting Your Crawlers
- Layer 2: Server Hardening, and the 403-vs-444 Question
- Layer 1: Blocking at the Edge, and Why We Do Not Challenge
- Why We Moved Scanner Blocking to the Edge, and What It Changed
- Freshness Signals: What AI Crawlers Actually Notice
- How to Read Your Own Server Log for AI Crawler Activity
- How We Audit a Site: The Checklist We Use
- What a Content-Signal Line Does, and What It Cannot Do
- What Happened When We Blocked Scanner Traffic at the Edge
- Why One Site's Crawl Numbers Are Not a Benchmark
- How AI Answer Engines Choose Which Pages to Cite
- How Googlebot Decides What to Crawl First
- Tools overview
- AI Access Badge
- AI Policy File Builder
- Answer Block Builder
- Edge vs Origin Check
- AI Crawler Log Analyzer
- AI Crawler Radar
- Robots.txt AI Access Check
- Schema.org JSON-LD Generator
- Structured Data Linter
- Step & Standards Block Builder
- AI Visibility Prompt Tester
- Visual Metadata Builder
- Content Workload Planner
- Privacy Policy
- Terms of Service
- Cookie Policy
- Disclaimer
- Site map
The machine-readable index is at /sitemap.xml, and the AI-facing summary at /llms.txt.
