Most robots.txt bugs are not typos. They are correct-looking files that a person reads top to bottom and a parser reads by specificity. If you believe the first matching rule wins, or that a Disallow beats an Allow because it sounds stricter, you will eventually block something you meant to open — or open something you meant to block — and the file will look fine in review.
Google open-sourced its production robots.txt parser in 2019, and the Robots Exclusion Protocol was later standardized as RFC 9309. Between those two things, the matching rules are no longer folklore. Here is what the parser actually does, and where third-party crawlers diverge from it.
Only one group applies to a crawler, and picking the wrong one is the most common bug
A robots.txt file is a set of groups. Each group is one or more User-agent lines followed by rules. Google's documentation is explicit: a crawler selects the single most specific group that matches its token, and ignores every other group entirely. Groups with the same user-agent token are merged, but groups with different tokens are not.
That means this file does not do what its author intended:
User-agent: *
Disallow: /internal/
Disallow: /cart
User-agent: Googlebot
Allow: /internal/docs/Googlebot matches the second group, so the second group is the only one it reads. /internal/ and /cart are not disallowed for Googlebot at all — those rules live in a group Googlebot never evaluates. If you want a crawler-specific exception, you have to restate every rule inside that crawler's group.
User-agent tokens are matched case-insensitively, and product-specific crawlers fall back to the generic token when no group names them. If there is no Googlebot-Image group, Google Images uses the Googlebot group; if there is no Googlebot group, it uses *.
Longest match wins, and ties go to allow
Within the selected group, order does not matter. Google compares the length of the rule's path string against the URL path and applies the most specific match. When two rules of equal specificity conflict, the least restrictive one — the Allow — wins.
| Rules in the group | URL path | Result | Why |
|---|---|---|---|
Allow: /p and Disallow: / | /page | Crawlable | /p is longer than / |
Allow: /folder and Disallow: /folder | /folder/page | Crawlable | Equal length, tie goes to allow |
Allow: / and Disallow: /*.php$ | /index.php | Blocked | The disallow path string is longer |
Disallow: /admin | /Admin/settings | Crawlable | Paths are case-sensitive |
Disallow: /search | /search-results | Blocked | Prefix match, no implied / boundary |
The last two rows account for a lot of real incidents. Rule paths are matched as prefixes and are case-sensitive, so Disallow: /search also blocks /searchable-inventory, and a rule written for /Category/ leaves /category/ wide open on a case-insensitive server.
This is also where third-party tooling diverges. Some crawlers and older libraries implement first-match-wins. If your site auditor says a URL is blocked and Search Console says it is crawlable, check the precedence model before you assume one of them is broken. Scrapy's Protego library explicitly aims for Google-compatible matching; a hand-rolled regex check almost certainly does not match it.
Wildcards are supported, but they are not regular expressions
Google supports exactly two special characters in rule paths:
*matches any sequence of characters, including none.$anchors the match to the end of the URL.
Everything else is literal. There are no character classes, no alternation, no negation, and no way to say "block this parameter unless another parameter is present." A trailing / is a literal slash, not a directory concept.
Two consequences worth internalizing:
Parameter blocking is order-dependent in the URL, not in the file. Disallow: /*?sort= only matches when ?sort= appears; /shoes?color=red&sort=price needs Disallow: /*sort= or Disallow: /*&sort= to catch. If your faceted URLs can emit parameters in any order, write rules that do not assume a position.
$ changes specificity, not just matching. Disallow: /*.pdf$ blocks /report.pdf but not /report.pdf?download=1. That is often the behavior you want, and often not.
An empty Disallow: line means "nothing is disallowed." Crawl-delay is parsed by some crawlers, including Bing, but Google ignores it. Sitemap is independent of groups and applies file-wide.
What happens when robots.txt returns an error
This is the part that turns a five-minute server hiccup into a crawling incident, and it is documented behavior rather than inference:
- 2xx: the file is fetched and parsed. Google generally caches it for up to 24 hours and honors
max-agecaching headers within its own limits. - 3xx: Google follows up to five redirect hops, then treats the result as a 404. Meta refresh and JavaScript redirects are not followed for robots.txt.
- 4xx other than 429: treated as "no robots.txt exists." Nothing is disallowed.
- 429 and 5xx: treated as a full disallow while the error persists. Google retries, and if the file stays unreachable for more than 30 days it falls back to the last cached copy, or assumes no restrictions if there is no cache. Google also notes that if it can determine a site is misconfigured to return 5xx for missing files, it treats the response as a 404.
So a robots.txt that 503s under load is temporarily equivalent to Disallow: /. If your robots.txt is generated by the application layer, it inherits every outage the application has. Serve it statically or from the edge with a hard-coded fallback.
Google also enforces a 500 kibibyte limit. Content past that point is ignored, which matters for auto-generated files that append per-product exclusions.
Scope: origin, not domain
A robots.txt file governs exactly one origin — scheme, host, and port. https://example.com/robots.txt says nothing about http://example.com, https://www.example.com, https://cdn.example.com, or https://example.com:8443. Every subdomain that serves crawlable content needs its own file. Redirecting https://sub.example.com/robots.txt to the apex file works because Google follows the redirect and applies the fetched rules to the requesting host, but that is a deliberate choice, not a default.
Also: noindex in robots.txt has been unsupported by Google since September 2019. If you still have those lines, they are inert comments.
What to do
- Audit group selection first. For every crawler-specific group in your file, confirm the rules that apply to
*are duplicated inside it. This single check catches more real exposure than any wildcard review. - Test with a Google-compatible parser, in CI. Build a fixture list of URLs that must be crawlable (canonical templates, key landing pages, JS and CSS bundles) and URLs that must be blocked (internal search, cart, faceted combinations). Run them through the open-source
google/robotstxtparser or Protego on every change to the file. - Stop relying on order for intent. If a reviewer needs order to understand the file, add comments explaining which rule wins by length. The parser will not read them, but the next engineer will.
- Serve robots.txt from static hosting or the CDN edge, with a cached fallback, so a 5xx on your app tier cannot become a site-wide disallow.
- Check the Search Console robots.txt report after any change. It shows the fetched file, the HTTP status, and the fetch time — which is how you confirm Google has actually picked up your edit rather than serving a cached copy for another few hours.
- Normalize case and boundaries. Prefer rules that match the exact path segment you mean (
/search/over/search) unless prefix bleed is intentional, and make sure your server does not serve the same content under multiple casings.