robots.txt Rules Match by Length, Not by Order

Most people read a robots.txt file top to bottom and assume the crawler does too — first matching rule wins, later rules override earlier ones, the User-agent: * block sets defaults that named bots inherit. All three of those assumptions are wrong for Google, and each one produces a different class of silent bug.

The actual semantics are specified in RFC 9309, which Google co-authored, and implemented in the robots.txt parser Google open-sourced in 2019. Three rules do most of the work: one group applies per crawler, the longest matching path pattern wins, and ties go to Allow.

Why the longest pattern wins instead of the first one

When Google evaluates a URL, it collects every Allow and Disallow line in the applicable group whose path pattern matches, then picks the one with the longest path pattern. Order in the file is irrelevant. If an Allow and a Disallow match with equal length, the least restrictive rule — Allow — wins.

Matching runs against the path and query string, it is case-sensitive, * matches any sequence of characters, and $ anchors the end of the URL.

Take this file:

User-agent: *
Disallow: /products/
Allow: /products/*/reviews
Disallow: /*?sort=
Allow: /$

That last row is the one that bites. A Disallow: /products/ does nothing for /Products/, and if your CMS serves both casings you have a crawlable duplicate set you never intended.

The practical consequence of length-based matching: you cannot "re-block" something by adding a Disallow further down the file. If an earlier Allow has a longer pattern, it still wins. To narrow an exception you have to make the blocking pattern longer and more specific than the allowing one.

One group applies, and named bots do not inherit from the wildcard

Google selects exactly one group of rules for a crawler: the group whose user-agent token is the most specific match for that crawler's name. If a Googlebot group exists, the User-agent: * group is ignored entirely for Googlebot — not merged, not used as a fallback for paths the specific group doesn't mention.

This configuration is a common accident:

User-agent: *
Disallow: /admin/
Disallow: /checkout/
Disallow: /search

User-agent: Googlebot
Disallow: /staging/

Googlebot is allowed to crawl /admin/, /checkout/, and /search. The wildcard group never applies to it. Someone added a Googlebot-specific line years ago and quietly deleted every other restriction for the only crawler that mattered.

Two related details:

• User-agent tokens are case-insensitive and matched as prefixes of the product token, so googlebot and Googlebot are the same group. Matching is on the crawler's product token, not the full user-agent string — don't write Mozilla/5.0 (compatible; Googlebot/2.1...) as a user-agent line.
• Sub-crawlers fall back to their parent token. Google documents that if there is no group for Googlebot-Image, it follows the Googlebot group; if there is no Googlebot group either, it follows *.
• Multiple groups with the same token are merged into one group before evaluation.

What Google does when robots.txt errors, redirects, or times out

The failure modes are documented and they are not symmetrical. A missing file means "crawl everything". An erroring file means "crawl nothing".

Two operational implications. First, if your WAF or rate limiter returns 429 or 503 for /robots.txt under load, you are telling Google to stop crawling the whole host until it recovers — which is sometimes exactly what you want during an incident, and catastrophic when it happens accidentally for a week. Second, serving an HTML error page with a 200 status for a missing robots.txt is worse than a clean 404: the parser will try to read it, ignore the unparseable lines, and apply whatever it happens to recognize.

Google also documents that it generally caches robots.txt for up to 24 hours, and may use a cached copy longer when refetching fails. Changes are not instant, so a robots.txt edit is not a same-hour lever.

One more limit: Google enforces a maximum of 500 kibibytes and ignores content past that point. Auto-generated robots.txt files that append a Disallow per filtered parameter can cross that line. Anything important belongs near the top.

Disallow is not noindex, and the two directives conflict

A disallowed URL can still appear in search results — typically without a snippet — if Google learns about it from links elsewhere. Blocking is about fetching, not about indexing.

The failure case is putting noindex in a page Google is not allowed to fetch. Google cannot see a meta tag or X-Robots-Tag on a URL it never requests, so the noindex has no effect and the URL can persist in the index indefinitely. If the goal is removal, allow the crawl and serve noindex. If the goal is saving crawl capacity on URLs you don't care about indexing, Disallow is correct.

How to test your file instead of reading it

Don't reason about robots.txt by eye, and don't trust an arbitrary library. Several popular parsers — including Python's standard-library urllib.robotparser — use first-match-in-file-order semantics and have limited wildcard support, which gives different answers than Google on exactly the cases that matter. Verify your parser's behavior against the RFC 9309 examples before you build tooling on it.

Use one of these instead:

• Google's open-source C++ robots.txt parser, which is the reference implementation of Googlebot's matcher.
• protego (the parser Scrapy uses), which targets Google-compatible longest-match semantics.

Then turn it into a regression test rather than a one-off check:

from protego import Protego

rp = Protego.parse(open("robots.txt").read())

cases = [
    ("https://example.com/products/123/reviews", True),
    ("https://example.com/products/123?sort=price", False),
    ("https://example.com/checkout/", False),
]
for url, expected in cases:
    got = rp.can_fetch(url, "Googlebot")
    assert got == expected, f"{url}: expected {expected}, got {got}"

Better still, feed it real URLs: pull a sample from your access logs and your XML sitemaps, run both through the parser, and flag two things — sitemap URLs that come back blocked, and blocked URL patterns that are nevertheless receiving verified Googlebot hits (which usually means your deployed robots.txt differs from the one in your repo).

What to do this week

1. Fetch your live robots.txt and check the status code, content type, and byte size. A 200 with text/plain under 500 KiB is the only safe combination.
2. Search the file for any named user-agent group. For each one, confirm it repeats every restriction from the * group that should still apply.
3. Re-evaluate every Allow exception by pattern length, not by position, and lengthen the blocking pattern wherever an exception is wider than you intended.
4. Check what your origin and CDN return for /robots.txt during a rate-limit or maintenance event. If the answer is 429 or 503, decide deliberately whether you want that to halt crawling.
5. List every URL where you rely on noindex and confirm none of them are disallowed.
6. Commit the parser test above alongside the file so a future edit that widens access fails CI instead of failing in production.