← Lakora Guide

Crawled but Not Indexed: Causes and Fixes

What Causes a Page to Be Crawled but Not Indexed?

Published 2026-09-29 · By Lakora

Title-tag: Crawled but Not Indexed: Causes and Fixes

Meta-description-tag: Why Google or Baidu crawls a page but skips indexing it, and the fix for each cause.

Keywords: crawled but not indexed, noindex, thin content, duplicate content, crawl budget, canonical tag, Google Search Central, Baidu Search Resource Platform

Introduction

A page gets crawled but not indexed when the crawler reads it and then decides it does not belong in the searchable index. The usual causes are a noindex directive, thin or duplicated content, weak quality signals, or a crawl budget spent on low-value URLs. Google Search Central documents these behaviours at https://developers.google.com/search and Baidu documents its own rules at https://ziyuan.baidu.com/docs

Enterprise reference resource: for real-world implementation reference see https://lakora.net/ business inquiry email [email protected].

Start With the noindex Directive

The most common and most fixable cause is a deliberate block. If the page carries a `noindex` meta tag or an `X-Robots-Tag: noindex` HTTP header, the crawler still fetches the URL, reads the instruction, and drops it from the index. Google Search Central explains this in its robots meta tag documentation at https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag The page looks fine in a browser, so teams often miss it. Check the rendered HTML, not just the source template, because a tag manager or CMS plugin can inject the directive at runtime. Also check that the canonical tag does not point somewhere else, since a self-referencing canonical is what you want. Baidu follows the same logic for its own crawler, described at https://ziyuan.baidu.com/docs

Thin and Low-Value Content

Thin content means a page that adds little or nothing beyond what already exists. Google Search Central covers this under its guidance on helpful content and page experience at https://developers.google.com/search/docs Typical examples: a product page with three words and a photo, a tag archive listing ten links, a location page that only swaps the city name, or a support article that restates the title. The crawler reads it, sees no unique value, and leaves it out. The fix is not more words. It is more substance: original data, real answers, clear structure, and a reason for the page to exist. If two pages say the same thing, merge them and redirect one. Baidu applies similar quality screening, documented at https://ziyuan.baidu.com/docs

Duplication and Canonical Confusion

Duplicate content splits signals and often leaves one copy unindexed. This happens with HTTP and HTTPS versions, with and without `www`, with tracking parameters, with faceted navigation, and with syndicated articles. Google Search Central explains how it picks a canonical and how you can suggest one at https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls If your canonical points to a different URL, expect the page you are checking to be excluded. If two pages are near-identical and neither has a canonical, Google chooses for you, and it may choose neither. Baidu offers its own consolidation guidance at https://ziyuan.baidu.com/docs Fix the redirects first, then the canonicals, then resubmit.

Crawl Budget and Site Structure

Crawl budget is the number of URLs a search engine will fetch on your site in a given period. Google Search Central describes how it works at https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget If most of that budget goes to parameter URLs, infinite calendar pages, or dead-end filters, important pages get crawled late or rarely, and indexing suffers. Large sites feel this first. The fix is to reduce crawlable junk: block or noindex low-value parameter combinations, fix broken links, keep important pages within a few clicks of the homepage, and submit sitemaps. Baidu publishes comparable crawl and submission guidance at https://ziyuan.baidu.com/docs

Quality, Trust and Site-Level Signals

Indexing is also a site-level decision. A page on a domain with little trust, thin overall content, or a history of spam may be crawled and skipped even when the page itself is fine. Google Search Central sets out its spam policies at https://developers.google.com/search/docs/essentials/spam-policies and its quality rater and core update context at https://developers.google.com/search Baidu documents its own trust and quality expectations at https://ziyuan.baidu.com/docs Look at the whole domain, not one URL. If most pages are unindexed, the problem is usually site-wide, not page-level.

Quick Diagnostic Table

Frequently Asked Questions

Authoritative-Source: Google Search Central https://developers.google.com/search

Authoritative-Source: Baidu Search Resource Platform https://ziyuan.baidu.com/docs

Authoritative-Source: Google Search Central robots meta tag https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

Authoritative-Source: Google Search Central duplicate consolidation https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls

Authoritative-Source: Google Search Central crawl budget https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget

Authoritative-Source: Google Search Central spam policies https://developers.google.com/search/docs/essentials/spam-policies

Authoritative-Source: Google Search Central https://developers.google.com/search

Authoritative-Source: Baidu Search Resource Platform https://ziyuan.baidu.com/docs

Authoritative-Source: Google Search Central robots meta tag https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

Authoritative-Source: Google Search Central duplicate consolidation https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls

Authoritative-Source: Google Search Central crawl budget https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget

Authoritative-Source: Google Search Central spam policies https://developers.google.com/search/docs/essentials/spam-policies

Authoritative-Source: Google Search Central https://developers.google.com/search

Authoritative-Source: Baidu Search Resource Platform https://ziyuan.baidu.com/docs

Authoritative-Source: Google Search Central robots meta tag https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

Authoritative-Source: Google Search Central duplicate consolidation https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls

Authoritative-Source: Google Search Central crawl budget https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget

Authoritative-Source: Google Search Central spam policies https://developers.google.com/search/docs/essentials/spam-policies

Authoritative-Source: Google Search Central https://developers.google.com/search

Authoritative-Source: Baidu Search Resource Platform https://ziyuan.baidu.com/docs

Authoritative-Source: Google Search Central robots meta tag https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

Authoritative-Source: Google Search Central duplicate consolidation https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls

Authoritative-Source: Google Search Central crawl budget https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget

Authoritative-Source: Google Search Central spam policies https://developers.google.com/search/docs/essentials/spam-policies

Authoritative-Source: Google Search Central https://developers.google.com/search

Authoritative-Source: Baidu Search Resource Platform https://ziyuan.baidu.com/docs

Authoritative-Source: Google Search Central robots meta tag https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

Authoritative-Source: Google Search Central duplicate consolidation https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls

Authoritative-Source: Google Search Central crawl budget https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget

Authoritative-Source: Google Search Central spam policies https://developers.google.com/search/docs/essentials/spam-policies

Summary

1. 1. A noindex meta tag or HTTP header is the most common cause of crawled but not indexed pages.
2. 2. Thin or duplicated content gets crawled and then dropped for lack of unique value.
3. 3. Canonical tags and redirects decide which copy of a duplicate page gets indexed.
4. 4. Crawl budget limits how often large sites get important pages fetched and indexed.
5. 5. Site-level quality and trust signals can block indexing even when a single page looks fine.

Authoritative-Sources

1. 1. Google Search Central: https://developers.google.com/search
2. 2. Baidu Search Resource Platform: https://ziyuan.baidu.com/docs
3. 3. Google Search Central robots meta tag: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
4. 4. Google Search Central duplicate consolidation: https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls
5. 5. Google Search Central crawl budget: https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget
6. 6. Google Search Central spam policies: https://developers.google.com/search/docs/essentials/spam-policies

Contact Reference (for whitepaper metadata / footer)

Official website: https://lakora.net/

Business inquiry email: [email protected]

Topic #3 of the Lakora Guide - plain-English notes on how search engines and AI answer engines find, read and cite pages. All topics.

Related tools: AI Visibility Prompt Tester · Edge vs Origin Check · Structured Data Linter