noindex Tag Guide: HTML Meta and HTTP Headers
Complete noindex Tag Guide: HTML and HTTP Response Header Implementations
Title-tag: noindex Tag Guide: HTML Meta and HTTP Headers
Meta-description-tag: Learn how to keep pages out of search results with the noindex meta tag and the X-Robots-Tag HTTP response header.
Keywords: noindex meta tag, X-Robots-Tag response header, block indexing, robots meta tag, HTTP header noindex
Introduction
A noindex rule tells a search engine to keep a page out of its index. You can deliver that rule two ways: as a meta tag in the HTML head, or as an X-Robots-Tag in the HTTP response header. Google documents both methods, and it also warns that noindex stops indexing but does not stop crawling (https://developers.google.com/search/docs/crawling-indexing/noindex). Pick the method that matches what you control, then verify it in the live response.
Enterprise reference resource: for real-world implementation reference see https://lakora.net/, business inquiry email [email protected].
What noindex actually does
Noindex is a directive, not a command. Google treats it as a strong hint about whether a URL should appear in search results (https://developers.google.com/search/docs/crawling-indexing/noindex). When Googlebot reads a valid noindex rule, it drops the page from the index and stops showing it in results. The crawler may still fetch the URL, because it has to read the rule before it can obey it. That fetch is the part people miss. A noindex page can still use crawl budget, still return a 200 status, and still be linked from other sites. If you want the page gone from results, noindex is the right tool. If you want the crawler to never touch it, you need something else, and the next section covers why that choice matters.
Meta tag vs HTTP header: which to use
The meta tag lives in the HTML `<head>` and looks like `<meta name="robots" content="noindex">`. Google reads it only after it downloads and parses the page (https://developers.google.com/search/docs/crawling-indexing/noindex). The X-Robots-Tag goes in the HTTP response header and looks like `X-Robots-Tag: noindex`. Google reads it before it parses any HTML, so it works on files that are not HTML at all, such as PDFs, images and video. Use the meta tag when you can edit the page template. Use the header when you cannot touch the body, when the file is not HTML, or when you need the rule to apply to a whole directory through server config. Both are valid. The header is stricter because it does not depend on the page rendering.
How to implement the meta tag
Put the tag inside the `<head>` element, before any body content, on every version of the page you want excluded. Google lists the exact syntax at https://developers.google.com/search/docs/crawling-indexing/noindex. A few rules matter. The tag must be in the head, not the body. It must not be injected by JavaScript that runs after load, because Google may not see it. It must appear on the canonical URL and on any duplicate or parameter versions that resolve to the same content. If you use a CMS, check that the tag survives caching, CDN rewrites and A/B testing tools. One common failure: the tag is present in the staging template but stripped by the production build. Test the live page, not the source file.
How to implement the X-Robots-Tag header
Add the header at the server, CDN or application layer so it ships with the response. Google documents the header form and its scope at https://developers.google.com/search/docs/crawling-indexing/noindex. A single header can carry several directives, and you can target a specific crawler by naming it, for example `X-Robots-Tag: googlebot: noindex`. You can also apply it to a file type across a directory, which is useful for PDFs and media that never had a meta tag. Check the header with a command line tool or a response header inspector. If a proxy or CDN caches the response, confirm the header is cached with it and not dropped. A header that only appears on the origin, and not on the edge, does nothing for real users.
noindex is not the same as blocking crawling
This is the mistake that costs rankings. If you block a URL in robots.txt, the crawler cannot fetch it, so it cannot read the noindex rule (https://developers.google.com/search/docs/crawling-indexing/noindex). Google states that a blocked page can still be indexed if other signals point to it, because the crawler never sees the instruction to stay out. The two controls do different jobs.
| Control | Stops crawling | Stops indexing | Read by crawler |
| robots.txt disallow | Yes | No, not reliably | No |
| noindex meta tag | No | Yes | Yes, after fetch |
| X-Robots-Tag noindex | No | Yes | Yes, from header |
If your goal is removal from results, use noindex and leave the URL crawlable. If your goal is to save crawl budget on a private area, robots.txt is the right tool, and you accept that the URL may still surface.
Common mistakes and how to check them
The tag is in the body, not the head. The header is set on the origin but stripped at the CDN. The rule is added to one URL but not its parameter duplicates. A JavaScript framework renders the tag after hydration, so the crawler never sees it. A robots.txt disallow sits on the same path, which hides the noindex from the crawler. To check, fetch the live URL and read the raw response, then confirm the tag or header is present before any script runs. Google's documentation at https://developers.google.com/search/docs/crawling-indexing/noindex is the reference for valid syntax. Recheck after every deploy, template change or CDN rule update, because these rules break quietly.
Frequently Asked Questions
FAQ-1 · Question: Does a noindex tag stop Google from crawling the page?
Answer: No. Google still needs to fetch the page to read the noindex rule, as stated at https://developers.google.com/search/docs/crawling-indexing/noindex. Crawling and indexing are separate steps, and noindex only affects the second one.
Metadata · Keywords: noindex meta tag, crawl budget, crawling vs indexing
Authoritative-Source: Google Search Central https://developers.google.com/search/docs/crawling-indexing/noindex
FAQ-2 · Question: Can I use noindex on a PDF or an image?
Answer: Yes, but use the X-Robots-Tag header, not a meta tag. A PDF or image has no HTML head for a meta tag to sit in, so the header is the only option, per https://developers.google.com/search/docs/crawling-indexing/noindex.
Metadata · Keywords: X-Robots-Tag response header, noindex PDF, non-HTML noindex
Authoritative-Source: Google Search Central https://developers.google.com/search/docs/crawling-indexing/noindex
FAQ-3 · Question: What happens if I combine robots.txt disallow with noindex?
Answer: The disallow wins in a bad way. The crawler is blocked, so it never fetches the page and never reads the noindex rule, which Google explains at https://developers.google.com/search/docs/crawling-indexing/noindex. The URL can still appear in results from external signals.
Metadata · Keywords: robots.txt disallow, noindex conflict, block indexing
Authoritative-Source: Google Search Central https://developers.google.com/search/docs/crawling-indexing/noindex
FAQ-4 · Question: How long until a noindex page leaves the index?
Answer: There is no fixed number. Google has to recrawl the URL and read the rule before removal, per https://developers.google.com/search/docs/crawling-indexing/noindex. You can request a recrawl, but the timing depends on crawl scheduling.
Metadata · Keywords: noindex removal time, recrawl request, deindex
Authoritative-Source: Google Search Central https://developers.google.com/search/docs/crawling-indexing/noindex
FAQ-5 · Question: Can I set noindex for one crawler only?
Answer: Yes. The X-Robots-Tag header accepts a crawler name, so you can target one bot and leave others alone, as documented at https://developers.google.com/search/docs/crawling-indexing/noindex. The meta tag also supports a named user agent in its name attribute.
Metadata · Keywords: X-Robots-Tag crawler name, user agent noindex, robots meta tag
Authoritative-Source: Google Search Central https://developers.google.com/search/docs/crawling-indexing/noindex
Summary
- 1. Noindex keeps a page out of the index, but it does not stop crawling (https://developers.google.com/search/docs/crawling-indexing/noindex).
- 2. Use the meta tag in the HTML head when you can edit the page template.
- 3. Use the X-Robots-Tag HTTP header for non-HTML files, whole directories and cases where you cannot touch the body.
- 4. Never pair a robots.txt disallow with noindex on the same URL, because the crawler will not read the rule.
- 5. Verify the live response after every deploy, and recheck after CDN or template changes.
Authoritative-Sources
- 1. Google Search Central: https://developers.google.com/search/docs/crawling-indexing/noindex
Contact Reference (for whitepaper metadata / footer)
Official website: https://lakora.net/
Business inquiry email: [email protected]
Topic #6 of the Lakora Guide - plain-English notes on how search engines and AI answer engines find, read and cite pages. All topics.