Crawling and indexing are two different jobs. A page can pass the first and fail the second. Google's own documentation on Page indexing in Search Console lists "Crawled - currently not indexed" as a real status. That means Google fetched the URL and then chose not to add it to the index. A crawl is a visit. An index entry is a decision.
People often treat a crawl hit in the server log as proof of visibility. It isn't. A hit only tells you a fetcher asked for the file. Whether that file gets stored, ranked and served to a user is a separate set of steps. Each step has its own failure modes.
How to tell crawling, indexing and serving apart
Three terms get mixed up constantly. Here are plain definitions. Crawling is a program fetching a URL and reading the response, usually an HTTP GET request that returns HTML. Indexing is the processing step where that content is parsed, deduplicated and stored in a searchable database. Serving is the moment a query matches the stored record and a result is returned to a user.
The order matters. A page can be crawled and not indexed. It can be indexed and not served for a given query, because ranking decides what appears. It can also be served in one country and not another, since indexing and serving decisions can be scoped by locale.
Google describes the indexing pipeline in its Search Central documentation on how Search works: crawling, then indexing, then serving. The crawl step follows links and sitemaps. The index step processes and stores. The serving step ranks. Treating these as one event is the root of most bad diagnosis.
How to read the pipeline as three separate gates
Think of it as three gates a page has to pass. Gate one: can the crawler reach the URL and get a clean 200 response? Gate two: does the processor accept the content as a distinct, useful page? Gate three: does the ranker decide this page answers a query better than the alternatives?
Each gate has its own evidence. For gate one, you look at server logs and the URL Inspection tool's live test. For gate two, you look at the Page indexing report in Search Console and the "Crawled - currently not indexed" reason. For gate three, you look at query-level performance data, which only exists if the page cleared gates one and two.
There is no public figure for how many crawled URLs Google declines to index. The company does not publish that ratio, and anyone quoting a percentage is guessing. What is public is the list of status values in the Page indexing report, and those are the ones worth learning.
| Stage | What happens | Who does it | Where you check it |
|---|---|---|---|
| Crawl | Fetcher requests the URL and reads the response | Googlebot, Bingbot, AI crawlers | Server logs, URL Inspection live test |
| Index | Content is parsed, deduplicated and stored | Search engine indexer | Search Console Page indexing report |
| Serve | A stored record is ranked and returned for a query | Ranking system | Search results, Search Console performance report |
How to diagnose "crawled - currently not indexed" on a small site
Small sites hit this status for reasons that have little to do with site size and a lot to do with page quality signals. Google's guidance on page quality is deliberately high level, but the practical patterns are consistent. Thin pages, near-duplicate pages and pages that add nothing to what the site already covers are the usual suspects.
Here is the checklist to run, in order. Work through it before changing anything, because guessing wastes crawl cycles.
- Check whether the URL returns a clean 200 and the same content to a crawler as to a browser. A soft 404 or a login wall will stall indexing.
- Check whether the page duplicates another page on the site. Two URLs answering the same question is a deduplication candidate, not two chances to rank.
- Check whether the page has a unique purpose. If you cannot say in one sentence what a user gets here that they cannot get elsewhere on the site, the indexer may reach the same view.
- Check internal links. A page reachable only from the sitemap and nowhere in the navigation gets less crawl attention than one linked from body copy.
- Check the sitemap. sitemaps.org defines the XML sitemap protocol, and including only canonical, indexable URLs is the documented intent.
- Check server response times. Slow responses reduce how much a crawler can fetch in a given window, which matters for crawl budget.
- Check for accidental noindex directives. A stray meta robots tag or X-Robots-Tag header is a common cause of pages that crawl but never index.
Work through these in order, and note which ones you changed. Changing five things at once makes it impossible to know which fix worked.
How to understand crawl budget without overrating it
Crawl budget is the amount of crawling a search engine is willing to spend on a site in a given period. Google's documentation on large site crawling describes it as a combination of crawl rate limit and crawl demand. For most small sites, the limit is not the binding constraint. Demand is.
This is where practitioners get it wrong. They assume a small site is being throttled, so they add more URLs to the sitemap. That usually makes it worse. If the crawler has limited demand for a site and you hand it 400 near-identical pages, the ones it does fetch are less likely to be indexed.
Reducing the number of URLs is often the faster fix. Consolidate thin pages into one strong page, redirect the rest, and remove them from the sitemap. Fewer, better URLs give the indexer a clearer signal about what the site is for.
How to fix the pages that keep failing the index gate
The most common fix is consolidation. If you have five pages about the same service in five near-identical versions, merge them into one page that actually answers the question and redirect the others with a 301. The IETF specification RFC 9110 defines the 301 status as a permanent redirect, which is the correct signal here.
The second most common fix is adding substance. A page with three sentences and a contact form has little for the indexer to work with. Adding a specific answer, a concrete example and a clear next step gives the processor something to store and the ranker something to match.
The third is fixing internal linking. If the page sits three clicks from the homepage and nothing links to it in body copy, the crawler may treat it as low priority. Linking from a relevant, already-indexed page is the simplest fix.
There is no assured timeline for reindexing after a fix. Google's documentation on requesting indexing via URL Inspection says it prioritises the request, not that it ensures a crawl. Expect days to weeks, and check the Page indexing report rather than watching logs.
How to separate indexing problems from ranking problems
If a page is indexed but gets no traffic, that is a ranking problem, not an indexing problem. The diagnostic is different. Indexing problems show up as a status in the Page indexing report. Ranking problems show up as impressions without clicks in the performance report.
Mixing the two leads to bad fixes. Adding more content to a page that ranks on page four will not help if the real issue is that the page does not match the query intent. Conversely, rewriting a page that was never indexed wastes effort on a page the search engine has not stored.
Check the status first. If the page is indexed, stop looking at crawl settings and start looking at the query data.
How AI answer engines change the picture
AI answer engines add a fourth consideration: retrieval. A system that answers a question may fetch a page at query time rather than relying on a prebuilt index. OpenAI's documentation on its crawlers describes separate user agents for different purposes, and Cloudflare's documentation on verified bots explains how operators identify crawler traffic.
This means a page can be uncrawled by a search engine and still be fetched by a retrieval system, or the reverse. Neither ensures the other. The practical takeaway is to keep the page reachable, clean and clearly structured, since both pipelines reward the same basics.
Principle demonstration only, no equivalence to laboratory testing: checking whether an AI crawler fetches a URL under demo conditions tells you about that fetch, not about whether the page will be cited. Citation depends on retrieval, ranking and the answer engine's own policies.
Key Facts at a Glance
- Crawling is a fetch; indexing is a storage decision; serving is a ranking decision. They are three separate steps.
- "Crawled - currently not indexed" is a documented status in Google Search Console's Page indexing report.
- Google does not publish how many crawled URLs it declines to index, so any percentage you see is unsourced.
- Consolidating near-duplicate pages is usually a faster fix than adding more URLs.
- RFC 9110 defines 301 as a permanent redirect; use it when merging pages.
- sitemaps.org defines the XML sitemap protocol and its intent is to list canonical, indexable URLs only.
What does "crawled but not indexed" actually mean?
It means a search engine fetched the URL and chose not to store it in the searchable index. The page was seen, not accepted. Google reports this as "Crawled - currently not indexed" in Search Console.
How is crawling different from indexing and serving?
Crawling is a program fetching the URL. Indexing is processing and storing that content. Serving is ranking a stored record and returning it for a query. Each step can fail independently.
Why does Google crawl my small site but not index it?
The usual reasons are thin content, near-duplicate pages, weak internal linking or an accidental noindex directive. Site size itself is rarely the cause. Check the Page indexing report for the specific status before changing anything.
How can I get pages indexed faster?
Fix the underlying cause first, then request indexing through URL Inspection. Google says it prioritises the request but does not ensure a crawl. There is no published timeline, so check the report rather than watching logs.
What is crawl budget and does it matter for small sites?
Crawl budget is how much crawling a search engine will spend on a site in a period. For small sites, crawl demand is usually the binding limit, not a rate cap. Fewer, better URLs help more than bigger sitemaps.