Three freshness signals get treated as one: a feed, a sitemap timestamp, and a date on the page. They answer three different questions, and mixing them up is how sites end up telling crawlers things they did not mean.

Freshness is judged from dates and change history, not from edits that change nothing.

Three signals, three jobs

  • A feed says: here is what is new since you last looked. It is a notification channel, and it is only as good as its ordering.
  • Sitemap lastmod says: this URL changed at this time. It is a claim about the page, and a claim you can falsify by touching it for no reason.
  • A visible date plus structured dates say: a human can see when this was written, and so can a parser. If the two disagree, the page is lying to one of its audiences.

What we do on this site

  • Articles appear in the feed in publication order, with a stable GUID, so an aggregator never sees the same item twice.
  • Sitemap lastmod is refreshed on a schedule and only moves when a page is rewritten, not when a template changes.
  • Every content page carries a visible line (“Published … · By …”) that matches datePublished in its structured data.
  • The weekly reports are frozen once published: a dated record that changes after the fact is not a record.

What we can measure, and what we cannot

We can measure that a crawler fetched a URL, when, and with what result. We cannot measure whether a feed shortened the interval before the next fetch, because engines do not publish their scheduling. Treat freshness work as improving the odds, not as a lever you can pull on a dashboard.

Running it on your own site

  • Fetch your feed and check that items are ordered and that GUIDs are stable across two fetches.
  • Compare sitemap lastmod with the actual last change of each page; if everything moved today, your generator is lying.
  • Check that the visible date and the structured date agree, including the format.
  • Finally, ask your access log whether anything fetches the feed at all. If nothing does, the feed is for humans and should say so.
Source: our own implementation and access log; see /feed.xml, /sitemap.xml and the dated records under /reports/.

Does an RSS feed help AI crawlers?

It can help them notice new URLs sooner, and it gives aggregators a clean list. It does not replace a sitemap, and it does not guarantee a fetch.

Should lastmod change when I edit a page?

Yes, if the content changed. If only the template changed, no. A lastmod that moves for everything teaches crawlers to ignore it.

How often should a small site update?

As often as it has something real to say. Freshness without substance is noise with a timestamp.

Related reading: Five Layers of Scanner Defence, Tested on Our Own Site · Layer 5: Closing the Loop in Public

More on this site: live signals · dated reports · how to verify us