WriteMySEO / Blog / What Googlebot Does When robots.txt Returns 500 or 403
SEO technology

What Googlebot Does When robots.txt Returns 500 or 403

A 5xx on robots.txt halts crawling of the whole origin. A 403 removes every rule you wrote. Here's the documented behavior and how to monitor it.

Most teams monitor whether their pages return 200. Almost nobody monitors whether /robots.txt returns 200 — and that single file controls whether Googlebot crawls the origin at all.

The two failure modes are asymmetric and both are bad. A server error on robots.txt makes Google treat the entire host as disallowed and stop crawling. A 403 makes Google treat the file as nonexistent and crawl everything, including the paths you disallowed. A WAF rule that changes 200 to 403 for an unrecognized user agent can therefore silently delete your entire crawl policy without a single line of your robots.txt changing.

What each status code on robots.txt actually means to Googlebot

Google documents the handling explicitly in its robots.txt specification, and the behavior matches RFC 9309, the Robots Exclusion Protocol standard. Summarized:

Response to /robots.txtHow Google treats it
2xx with a parseable bodyRules apply as written
3xx (up to five hops)Follows redirects; uses the final file's rules
3xx with no resolution in five hopsTreated as 404
4xx other than 429 (incl. 401, 403, 404, 410)No robots.txt exists — no crawl restrictions
429, 5xx, network timeout, DNS failureServer error — temporarily stop crawling the site

Two entries on that table are the ones that cause outages. The 4xx row includes 401 and 403, which feel like "access denied" to a human and read as "there are no rules here" to a crawler. The 5xx row includes transient infrastructure blips, which feel harmless and read as "do not crawl this host."

Google also documents one escape hatch on the error row: if it can determine that a site is misconfigured to return 5xx instead of 404 for missing files, it may treat the 5xx as a 404. Do not plan around that heuristic. You cannot see whether it fired, and it exists to rescue broken sites, not to license broken responses.

Why a 5xx on robots.txt stops crawling the whole origin

The logic is conservative by design. A crawler that cannot read the rules cannot know which URLs it is permitted to request, so the standard tells it to assume everything is off limits rather than risk fetching something the publisher forbade.

In practice Google softens this with caching. Google's documentation says robots.txt contents are generally cached for up to 24 hours, and may be cached longer when a fresh copy cannot be fetched. So a five-minute 503 during a deploy usually costs you nothing: the cached file is still valid. A multi-hour outage starts to bite as the cache expires and retries keep failing.

Google further documents that if robots.txt stays unreachable for more than 30 days, it falls back to the last cached copy, and if no cached copy exists, it assumes there are no crawl restrictions. That is the long tail of the behavior, not a safety net you want to reach — in between, you have a window of degraded or halted crawling, and crawl demand does not instantly snap back when the file recovers.

The deliberate version of this mechanism is real and useful: returning 503 on robots.txt is a documented way to pause crawling during an emergency. It is a tourniquet, not a configuration. Leave it on for weeks and you are gambling with how Google re-evaluates URLs it can no longer verify.

Why a 403 is more dangerous than a 500

A 5xx is loud. Crawling drops, Search Console's Crawl stats report shows robots.txt fetch failures, and somebody notices.

A 403 is silent and inverted. From the moment it starts, every Disallow you wrote is gone. The reasonable inference — not a Google-documented claim — is that the first thing to absorb the freed crawl capacity is whatever you disallowed because it was infinite or worthless: faceted filter combinations, internal search results, session-parameter URLs, calendar pages. You do not get a warning. You get a crawl stats graph that climbs while your important URLs get crawled no more often than before.

Note that robots.txt controls crawling, not indexing. A lost Disallow does not directly put those URLs in the index — but it does let Google fetch them, discover links in them, and evaluate them for indexing. Recovery is slower than the outage.

The usual cause is bot management. Rate limiting, geo-blocking, a JavaScript challenge, or an origin shield that treats /robots.txt like any other path. If your edge can serve a challenge page to a crawler, it can serve one for robots.txt.

robots.txt is per-origin, and redirects move the goalposts

One robots.txt governs exactly one scheme + host + port. https://example.com/robots.txt does not apply to https://www.example.com/, to http://example.com/, or to https://shop.example.com/. Every hostname that resolves and serves content needs its own correct file, including staging hosts that leaked into DNS.

Redirects complicate this. Google follows up to five hops for robots.txt and uses the rules in the file it finally reaches, including across hosts. Google also documents that it does not follow logical redirects — meta refresh, JavaScript, or frame-based redirects — for robots.txt. A protocol-upgrade redirect chain that loops or exceeds five hops ends as a 404, meaning no restrictions.

A quick check per origin:

curl -sSIL -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://example.com/robots.txt | grep -E 'HTTP/|location:'

Run it with and without the Googlebot user agent. If the status codes differ, your edge is filtering the file.

How to measure robots.txt availability properly

Search Console, robots.txt report. Shows the fetch status and last-fetched time per host that Google checked in the last 30 days. It is the authoritative view of what Google actually received, and it covers subdomains you may have forgotten.

Search Console, Crawl stats. Contains a robots.txt availability chart — the share of fetch attempts that failed. Any sustained nonzero failure rate is worth a ticket.

Your logs. Filter for requests to /robots.txt grouped by status code, user agent, and host, and verify Googlebot by reverse DNS rather than by UA string. You will typically see repeated daily fetches per origin; a gap or a status change is the earliest signal you have.

An external monitor that asserts more than uptime. Check status code, a content hash, and the presence of a known line — with the Googlebot user agent, from outside your network. A monitor that only checks 200 OK from an allowlisted IP will not catch a bot-management 403.

What to do

  1. Add /robots.txt to your edge's bypass list: no WAF challenge, no rate limit, no geo rule, no auth.
  2. Serve it as a static file or from cache with a short TTL, so an application-tier outage cannot turn it into a 500.
  3. Alert on any non-200 for robots.txt within minutes, and alert separately on content-hash changes so a bad deploy that ships a Disallow: / is caught as fast as an outage.
  4. Enumerate every live origin — apex, www, http, subdomains, country hosts, CDN hostnames — and monitor each one. The rules are per-origin; the monitoring must be too.
  5. Reserve 503 on robots.txt for deliberate, short, documented crawl pauses. Set a reminder to undo it.
  6. After any incident, check Crawl stats for the recovery curve rather than assuming crawl rate returns the moment the file does.
robots.txtcrawlinggooglebotmonitoring

WriteMySEO produces marketing content, not legal, medical, financial, or compliance advice. Figures cited reflect publicly reported industry data at time of writing and shift over time.

Get started

We write this well about your industry, every month.

AI-drafted, human-reviewed SEO content on a flat subscription. Blog posts, metadata, schema, and internal links, shipped on a monthly rhythm.

See plans

More from the blog