Half the deindexing tickets I've reviewed contain the same instruction: "block it in robots.txt and add a noindex." That combination does not double your protection. It cancels it. A Disallow rule prevents Google from fetching the page, and the noindex lives inside the page it can no longer fetch.
This is the single most consequential confusion in technical SEO because it fails silently. Nothing errors. The page just sits in the index, often with a blank description, and the fix you deployed is the reason it's still there.
The two directives answer completely different questions
Think of them as separate gates in sequence.
robots.txt answers "may I request this URL?" It is fetched before anything else, it applies per-host and per-protocol, and it is a crawling control. A disallowed URL is never requested, so its status code, its content, its canonical tag, its meta robots tag, its hreflang, and its structured data are all invisible to the crawler.
noindex answers "may I show this URL in results?" It ships in the page's HTML <head> or in the HTTP response headers as X-Robots-Tag. It is an indexing control, and it only works if the response is retrieved.
Google stopped supporting a noindex: directive inside robots.txt as of September 1, 2019 — it announced the deprecation of that and several other unsupported rules in July 2019. If you inherited a robots.txt with Noindex: /path lines in it, they have done nothing for years.
Why applying both leaves the URL indexed
Google documents this plainly: a page blocked by robots.txt can still appear in search results if Google discovers it through links from other pages. It has enough signal from the anchor text and the URL itself to consider the page a candidate. It just has no content, so you get a result with the URL and, historically, the note that no information is available.
So the failure sequence is:
- The URL is already indexed, or gets linked from somewhere.
- You add
Disallowto robots.txt. - You add
<meta name="robots" content="noindex">to the page. - Googlebot never requests the page again, so it never sees step 3.
- The URL stays in the index indefinitely, minus a snippet.
In Search Console this surfaces as Indexed, though blocked by robots.txt in the Pages report. That status is the diagnostic fingerprint of this exact mistake. Compare it to Excluded by 'noindex' tag, which means the system worked: Google fetched the page, read the directive, and dropped it.
There's a second-order cost. A blocked URL's rel=canonical is also unreadable, so if your dedupe strategy depends on canonical tags on parameterized URLs, blocking those URLs in robots.txt breaks consolidation instead of helping it.
The decision table
Start from what you actually want, not from which file you happen to have access to.
| Goal | Use | Do not use |
|---|---|---|
| Keep a URL out of search results | noindex (meta or X-Robots-Tag), crawlable | robots.txt Disallow |
| Remove an already-indexed URL | noindex, crawlable, until it drops | Disallow (freezes it in place) |
| Stop wasting crawl on infinite/low-value URL spaces (calendars, session params, filter combos) | robots.txt Disallow | noindex alone — it still costs a fetch |
| Both: never crawl and never index | noindex first, verify it drops from the index, then Disallow | Both at once |
| Genuinely private data | Authentication or removal from the server | robots.txt (it's a public file that advertises the path) |
| Permanently delete a page | 404 or 410 | noindex as a substitute for deletion |
| Emergency suppression, right now | Search Console Removals tool (temporary, roughly six months) plus a permanent fix | Removals tool alone |
| Block a PDF, image, or feed from results | X-Robots-Tag: noindex header | meta tag (no HTML <head> to put it in) |
The sequencing row is the one people skip. If you truly need a URL both uncrawled and unindexed, noindex has to be live and honored before you block crawling — otherwise you're back to the failure above.
Serving noindex when you can't edit the HTML
X-Robots-Tag is the underused half of this. It works on any response type and can be applied at the web server or CDN edge, which matters when your CMS templates are locked down.
Apache:
<FilesMatch "\.(pdf|docx)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>nginx:
location ~* \.(pdf|docx)$ {
add_header X-Robots-Tag "noindex";
}Verify with a request, not with a browser tab:
curl -sI https://example.com/whitepaper.pdf | grep -i x-robots-tagOne caution: if a page has both a meta tag and a header and they conflict, the more restrictive interpretation tends to win, and you will spend an afternoon figuring out which layer is emitting what. Pick one mechanism per URL pattern and document it.
Edge cases that trip up experienced people
noindex on a page you also want to pass link signals. Google's stated position is that noindex, follow collapses over time — once the page is dropped, its outgoing links stop being useful the way a live indexed page's links are. John Mueller has described long-term noindex pages as eventually behaving like noindex, nofollow. Treat noindex as a signal terminus, not a pass-through.
noindex plus canonical on the same URL. Contradictory instructions: canonical says "consolidate me into that URL," noindex says "drop me." Google has to pick one, and the outcome isn't reliably predictable. Choose the intent and remove the other tag.
Disallowing your own noindexed pages to "save crawl budget." Only do this after the URLs have left the index. Otherwise you're trading a small fetch cost for a permanent index artifact.
Blocking JS or CSS that the noindex depends on. If noindex is injected client-side and the script is disallowed, the directive may never render. Server-side or header-based is safer.
Staging environments. robots.txt on staging is not protection; it's an invitation with directions. Use HTTP auth.
What to do this week
- Open Search Console → Pages and filter for Indexed, though blocked by robots.txt. Every URL there is either a robots.txt rule that should be removed, or a page that needs
noindexplus crawl access. - Grep your robots.txt for
Noindex:and delete those lines. They've been inert since 2019 and they mask real intent. - For every path you're disallowing to control indexing, ask whether the goal is crawl reduction or index removal. Move the index-removal cases to
noindexand unblock them. - Add a CI check that asserts your key templates return no
noindexand no matchingDisallow— accidentalnoindexon production is the mirror-image failure, and it's usually a build-config leak from staging. - Confirm with
curland the URL Inspection tool's live test, which reports both the robots.txt verdict and the indexing directive separately. If the robots verdict is "blocked," the indexing directive line is meaningless.