The First 15MB Is All Googlebot Reads of Your HTML

Google's crawler documentation contains a limit most people have read once and never acted on: Googlebot crawls the first 15MB of an HTML file, and anything past that byte is not used for indexing. It is stated plainly in Google's crawler docs, not inferred.

Most teams dismiss it because 15MB of prose is absurd — roughly 2.5 million words. But HTML files are not prose. They are prose plus hydration state, plus inlined SVG, plus base64 images, plus a server-rendered component tree that repeats the same props three times. The pages that hit this ceiling are not long articles. They are applications.

Where the 15MB limit applies, and where it doesn't

Three details in Google's documentation change how you should think about the cap:

• It applies per file, not per page. The HTML document gets 15MB. Each referenced CSS and JavaScript file is fetched separately and is bound by the same limit independently. A page can legitimately pull down 40MB in total resources without truncating anything.
• It applies to uncompressed bytes. Google's documentation states the limit is applied to the uncompressed data. Serving Brotli or gzip reduces transfer time and crawl cost, but it does not buy you headroom against this specific limit. A 2MB gzipped response that decompresses to 18MB is truncated.
• It applies to the fetched file, not the rendered DOM. Content injected by JavaScript after load is not governed by the HTML file cap. The Web Rendering Service has its own constraints, but they are not this constraint, and Google has not published them in comparable detail.

That last point produces a counterintuitive case: a client-rendered page with a thin HTML shell is not at risk from the 15MB limit, while the server-rendered version of the same page — which is otherwise better for indexing — can be.

What a truncated HTML file looks like to a parser

Truncation is not graceful. The fetch stops mid-byte. If the cut lands inside a tag, inside an attribute value, or inside a <script type="application/ld+json"> block, the parser receives malformed markup and recovers however HTML error handling dictates — which usually means dropping the broken node and everything that depended on it.

The practical consequences, in rough order of how often they matter:

1. Structured data placed near the end of the document disappears. JSON-LD blocks injected after the main content, which is common in tag-manager-style deployments, are the single most likely casualty.
2. Trailing body content is never seen. Reviews, FAQs, related-product grids, and footer internal links — all common at the bottom of large templates — simply do not exist from the crawler's point of view.
3. Internal links in the truncated region do not pass anything. They are not nofollowed; they are not there. This can quietly strip discovery paths you believed were in place.

<head> content is almost never affected, so canonical tags, meta robots, and hreflang survive. That is why truncation rarely shows up as a dramatic indexing failure. It shows up as a page that is indexed, renders a title and description fine, and is mysteriously missing rich results or appears to have no internal links to its children.

Which pages actually hit it

The realistic offenders are structural, not editorial:

• Inlined hydration state. Frameworks that serialize the full data payload into the document (__NEXT_DATA__, Nuxt's __NUXT__, Redux preloaded state, Apollo cache dumps) often duplicate every field already rendered in the markup. On a category page with 500 products and nested variant data, this can be the majority of the file.
• Base64 data URIs. Inlining images to avoid requests adds roughly 33% overhead over the binary size and puts it all in the HTML budget.
• Inline SVG icon systems rendered per-row instead of referenced via <use>.
• Unbounded server-rendered lists. Infinite-scroll implementations that server-render everything loaded so far, or tables dumping full datasets.
• Comment threads and user-generated content rendered without a cap.

How to measure the bytes Googlebot actually sees

Fetch the page the way the crawler does and measure the decompressed size:

curl -sS --compressed \
  -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  -o /tmp/page.html https://example.com/some-page
wc -c /tmp/page.html

wc -c on the written file gives uncompressed bytes, which is the number that matters. The threshold is 15,728,640 bytes if you read 15MB as binary megabytes; treat anything above about 10MB as a problem worth fixing regardless of where the exact line sits.

Then check where your important markup lives in the byte stream. grep -b prints byte offsets:

grep -abo '"@type": *"Product"' /tmp/page.html | head -1
grep -abo '__NEXT_DATA__' /tmp/page.html | head -1

If a structured data block starts at byte 14,900,000, it is one template change away from vanishing.

To find what is consuming the file, measure the inlined payloads directly:

python3 - <<'EOF'
import re
html = open('/tmp/page.html', encoding='utf-8', errors='replace').read()
total = len(html.encode())
scripts = sum(len(m.encode()) for m in re.findall(r'<script[^>]*>(.*?)</script>', html, re.S))
b64 = sum(len(m.encode()) for m in re.findall(r'data:image/[^"\')]+', html))
print(f"total={total} inline_script={scripts} base64_img={b64}")
EOF

A content-to-markup ratio is a crude metric, but the ratio of inline script bytes to total bytes is not — it tells you precisely which subsystem to go after.

Crawl at scale, not one URL at a time

One page tells you nothing about a template. Pull Content-Length from a crawl of a few thousand URLs, but verify it against decompressed size on a sample, because Content-Length reports compressed bytes when encoding is applied. In server logs, the response-size field has the same caveat. The reliable approach is to sample 50–100 URLs per template, decompress, and record the distribution. Templates behave consistently; outliers within a template usually mean an unbounded list.

What to do

1. Measure uncompressed HTML size per template. Flag anything over 5MB for investigation and anything over 10MB for immediate work.
2. Move structured data into the <head> or as early in the body as your stack allows. This costs nothing and removes the highest-impact failure mode.
3. Prune hydration payloads. Serialize only the fields the client actually needs on first interaction. Most frameworks let you strip server-only fields before serialization.
4. Cap server-rendered list lengths and paginate with crawlable links rather than rendering the full set.
5. Replace base64 images and per-instance inline SVG with referenced files — each gets its own budget.
6. Re-measure after framework upgrades. Hydration payload shape changes between major versions, and nobody tracks HTML byte size in CI unless you add it. A single assertion in a smoke test is enough.