Sampling Index Coverage With the URL Inspection API

Most teams use the URL Inspection API the same way they used the UI: paste in a URL that looks broken, read the verdict, move on. That is a debugging tool, not a measurement tool. If you want to answer "what share of my product pages are indexed, and why aren't the rest?", one-off inspections will never get you there, and neither will the Page Indexing report, which gives you reasons in aggregate but won't tell you which segment they belong to.

The API can answer that question. It just can't answer it for every URL, ever, on a site of any size. The quota forces you to design a sample — and once you accept that, the interesting work is statistical rather than technical.

What the API actually returns, and what it doesn't

Google's URL Inspection API returns the indexed state of a URL — the same data as the "Google Index" section of the UI tool. It does not run a live test. There is no live-fetch equivalent in the API, so you cannot use it to check whether a fix you shipped ten minutes ago renders correctly.

The useful fields sit under inspectionResult.indexStatusResult:

{
  "indexStatusResult": {
    "verdict": "PASS",
    "coverageState": "Submitted and indexed",
    "robotsTxtState": "ALLOWED",
    "indexingState": "INDEXING_ALLOWED",
    "pageFetchState": "SUCCESSFUL",
    "googleCanonical": "https://example.com/p/123",
    "userCanonical": "https://example.com/p/123?variant=blue",
    "crawledAs": "MOBILE",
    "lastCrawlTime": "2024-11-02T09:14:31Z"
  }
}

Three of these are worth more than the verdict. googleCanonical versus userCanonical is the only scalable way to count canonical disagreements per segment. lastCrawlTime gives you a crawl-recency distribution without touching server logs. And coverageState is the string that actually names the reason — "Crawled - currently not indexed", "Discovered - currently not indexed", "Alternate page with proper canonical tag" — where the binary verdict just says FAIL.

The quota math: why full sweeps don't exist

Google documents the quota as 2,000 inspections per day and 600 per minute, per property, shared across everyone with access. That ceiling is the entire design constraint.

• 500,000 URLs at 2,000/day is 250 days for one pass. Your site will have changed several times over before the sweep finishes, so the "result" describes no single moment.
• Verifying additional URL-prefix properties for subdirectories is the common workaround, since the quota attaches to the property. Treat this as a practitioner tactic, not a documented entitlement, and check the current quota page before you build a pipeline that depends on it.
• Even at 10,000/day across five properties, a daily full sweep of a large catalog is out of reach.

The reasonable conclusion: stop trying to census the site. Sample it, and spend the saved quota on repeating the measurement.

How precise is "87% indexed"?

A simple random sample of n URLs estimates an indexation rate with a standard error of sqrt(p(1-p)/n). At p ≈ 0.85, the 95% margin of error looks like this:

Note what happens between 1,000 and 2,000: you double the spend to buy 0.6 points of precision. For most decisions, 1,000 inspections is past the point of diminishing returns on a single global number — which means a full day's quota is better spent on five strata of 400 than on one very precise sitewide figure you can't act on.

If you're sampling from a small stratum, apply the finite population correction: multiply the standard error by sqrt((N-n)/(N-1)). Inspecting 400 of 600 URLs in a category is far more precise than the table suggests.

Stratify by the thing you suspect

A sitewide indexation rate is almost useless, because indexation is not randomly distributed. It clusters by template, by depth, by whether the page has internal links, by whether it was in a sitemap. So define strata before you sample, from your own crawl data:

1. Template or path group — product, category, filter, editorial, pagination.
2. Internal inlink count, bucketed (0, 1–2, 3–10, 10+). This is usually the strongest predictor you control.
3. Click depth from the home page.
4. Publication or last-modified age.

Allocate roughly equal sample sizes per stratum rather than proportional ones. Proportional allocation gives your biggest template a tight estimate and leaves the suspicious long tail with 20 URLs and a ±20-point interval. Equal allocation gives every stratum the same decision quality, which is what you want when the whole point is to compare them.

When you need a single sitewide number afterwards, reweight: p_site = Σ (N_h / N) × p_h, where N_h is the true population of each stratum from your crawl. Do not average the stratum rates directly — that silently treats a 200-URL stratum as equal to a 200,000-URL one.

Re-inspect the same panel instead of redrawing it

The biggest power gain is free. If you draw a fresh random sample each month, comparing two 1,000-URL samples at p ≈ 0.85 gives a standard error on the difference of about 1.6 pp, so you can only detect shifts larger than roughly 3 points. Half your observed movement is resampling noise.

If instead you fix a panel of the same URLs and re-inspect them on a schedule, the comparison is paired. You throw away the URLs that didn't change state and test only the discordant ones — the McNemar setup. A panel where 30 URLs went from indexed to not-indexed and 8 went the other way is a clear signal, even though the headline rate moved barely a point.

Keep a small refresh slice — say 10% new URLs per cycle — so the panel doesn't age into irrelevance, and record it as a separate cohort rather than mixing it into the trend.

Read coverageState carefully

Two caveats that will otherwise produce false findings:

• The data lags. lastCrawlTime tells you how stale the record is. A URL you shipped yesterday will often return "URL is unknown to Google" simply because nothing has happened yet. Exclude URLs younger than your observed median discovery lag from indexation-rate calculations, or report them as a separate "too new to judge" bucket.
• "Alternate page with proper canonical tag" is not a failure. Bucket coverage states into intended non-indexing (canonicalized, noindex, redirect), blocked (robots, fetch errors), and genuinely withheld ("Crawled - currently not indexed", "Discovered - currently not indexed"). Only the third bucket is a quality signal worth chasing. Collapsing all three into "not indexed" is the single most common way these audits mislead.

What to run this week

1. Export your crawl, build four to six strata on template × inlink bucket, and record N_h for each.
2. Draw 300–400 URLs per stratum as a fixed panel, stored with a cohort ID.
3. Inspect the panel at 600/minute inside the daily cap, persisting the full indexStatusResult, not just the verdict.
4. Report per-stratum rates with intervals, a reweighted sitewide rate, and the canonical-mismatch count from googleCanonical vs userCanonical.
5. Re-inspect the same panel on a fixed cadence and report discordant transitions, not headline deltas.

That pipeline costs a fraction of your quota and produces a number you can defend when someone asks whether last quarter's internal linking change did anything.