Measuring Time-to-Index With the URL Inspection API

Ask how long Google takes to index a new page and you'll get an average from somebody's sample of somebody else's sites. That number is useless to you. Indexing latency is site-specific, section-specific, and often bimodal — a chunk of URLs picked up within hours and a long tail that sits for weeks. The average sits in the empty space between the two humps.

The good news: this is one of the few SEO metrics you can measure directly, on your own URLs, with a documented API. The bad news is that most implementations measure the wrong event and quietly bias the result.

What "time to index" actually means

There are four distinct events, and they can be days apart:

1. Discovery. Google learns the URL exists — from a sitemap, an internal link, an external link, a redirect target.
2. Crawl. Googlebot fetches it. Your logs see this.
3. Indexing. The page is processed and stored in the index, possibly after a rendering pass.
4. Serving. The URL becomes eligible to appear for queries, which is when it shows up in Search Console Performance data.

People say "indexed" and measure serving, or measure crawl and call it indexing. Decide which event your team cares about before you build anything. For publishing workflows, discovery→serving is usually the number that matters. For diagnosing a crawl problem, you want discovery→crawl isolated, because that's the interval a faster server or a better internal link actually moves.

The instrument: Search Console's URL Inspection API

The URL Inspection API returns, per URL, the same index status data the Search Console UI shows. Google documents a quota of 2,000 queries per day and 600 per minute per property. The fields that matter for latency work live under indexStatusResult:

• verdict — PASS, PARTIAL, FAIL, or NEUTRAL
• coverageState — the human-readable state string ("Submitted and indexed", "Crawled - currently not indexed", "Discovered - currently not indexed", "URL is unknown to Google")
• lastCrawlTime — timestamp of the most recent crawl
• robotsTxtState, indexingState, pageFetchState — why it isn't indexed, if it isn't
• googleCanonical vs userCanonical — whether Google agrees with your canonical
• sitemap and referringUrls — how Google found it

Two cautions. First, lastCrawlTime is the last crawl, not the first, so it stops being a proxy for "when did Google first fetch this" the moment a second crawl happens. Poll often enough that the first non-null value is close to the first crawl. Second, when Google selects a different canonical, the returned status describes the indexed canonical, not necessarily the URL you asked about — always compare googleCanonical to the inspected URL before recording a result.

The Indexing API is not an alternative here. Google restricts it to JobPosting and livestream BroadcastEvent pages; using it for general content submission is out of policy and gives you no measurement value.

Mapping states to stages

That distinction between Discovered and Crawled - currently not indexed is the single most useful output of the whole exercise. They look similar in a coverage report and have opposite fixes. Discovery-stage stalls respond to crawl-side work: sitemap hygiene, internal linking from frequently crawled pages, faster responses. Crawl-stage stalls do not — Google fetched the page and declined to index it, and no amount of resubmitting changes that judgment.

Designing the poll so it doesn't lie to you

You cannot measure this retroactively. Latency data only exists if you were watching.

# one cohort row per URL, appended daily until terminal state
for url in cohort_urls_not_yet_indexed():
    r = inspect(url)            # URL Inspection API
    s = r["inspectionResult"]["indexStatusResult"]
    record(url,
           checked_at=now_utc(),
           verdict=s["verdict"],
           state=s.get("coverageState"),
           last_crawl=s.get("lastCrawlTime"),
           g_canonical=s.get("googleCanonical"))

Design notes that change the answer:

• Define t₀ precisely. Publication time, or the first sitemap ping after publication? Pick one and use it forever. Mixing them makes cohorts incomparable.
• Poll daily, at a fixed hour. Daily polling gives you day-resolution latency, which is enough. Hourly polling burns quota and adds no decision value for most sites.
• Sample, don't census. With 2,000 calls/day, a 200-URL cohort tracked for 10 days fits comfortably. Stratify the sample by template and site section, because latency varies enormously between them.
• Censor honestly. URLs that never index are not missing data — they're the tail. Report them as "not indexed by day 30" rather than dropping them, which is exactly how you'd handle right-censored data in a survival analysis.
• Normalize timestamps to UTC before differencing.

Report the median and the 90th percentile, plus the share still unindexed at day 14 and day 30. Never report the mean.

Cross-checking with serving data

The Performance API gives you a second, independent instrument: the first date a URL received an impression. That measures serving, not indexing, and it's noisy — a page can be indexed for a week before any query surfaces it, and Search Console's anonymized-query filtering can suppress low-volume rows. But when inspection says a URL is indexed and Performance shows no impressions for weeks afterward, you've learned something different and more interesting: the page is in the index and losing every retrieval it enters. That's a relevance problem, not a crawl problem.

Log files give you the third instrument and the most reliable crawl timestamps — verified Googlebot fetches with exact times. Use logs for discovery→crawl, use the API for crawl→index, use Performance for index→serving.

What to do with the number

• Baseline before you change anything. Run one cohort of 100–200 URLs for 30 days and record the median, p90, and 30-day non-indexed share by template. Everything after this is a comparison against that baseline.
• Split the interval. If p90 discovery→crawl is long, work on sitemaps, internal links from high-crawl-rate pages, and server response time. If crawl→index is long or never completes, stop touching crawl levers and look at page quality, near-duplication, and canonical consolidation.
• Re-run cohorts after structural changes — a new hub page, a sitemap split, a template rewrite. A cohort comparison is the only way to know whether the change did anything.
• Set an internal SLO from your own distribution, not from an industry figure. "90% of new product pages indexed within 7 days" is a claim you can verify weekly.
• Alert on the shape, not single URLs. One slow page means nothing. A cohort whose p90 doubles month over month means crawl demand for your site has changed, and that's worth investigating before traffic reflects it.