WriteMySEO / Blog / AI Visibility Scores Need Repeated Sampling, Not One Run
SEO data

AI Visibility Scores Need Repeated Sampling, Not One Run

Answer engines are stochastic. Learn why a single prompt check tells you almost nothing, how many runs you actually need, and how to tell a real change from noise.

You ask ChatGPT a question and your brand gets cited. A colleague asks the same question twenty minutes later and it doesn't. Neither of you changed anything on the site.

That is not a bug in the tool or a sign that something broke. It is the expected behaviour of a stochastic system being sampled once. Most "AI visibility" reporting treats a single run as a measurement, when it is actually a single draw from a distribution nobody has estimated. Until you know the shape of that distribution, you cannot say whether your 38% mention rate went up, went down, or did nothing at all.

Where does the variance actually come from?

Four independent sources stack on top of each other, and only one of them is the model's decoding temperature.

Decoding randomness. Assistants sample tokens rather than taking the argmax. Different token choices early in the answer change which entities get named later. Even where a provider exposes a temperature of zero, exact reproducibility is not something hosted inference guarantees — batching, mixed precision, and expert routing in sparse models all make bit-identical outputs a non-goal. That is reasonable inference from how the systems are built, not a documented commitment either way.

Retrieval randomness. This is the bigger source. Google has described AI Mode as using a query fan-out technique: the system issues multiple related searches and synthesises across the results. The generated sub-queries are themselves model output, so they vary run to run. Different sub-queries hit different result sets, which means different candidate documents reach the generator. Approximate nearest-neighbour indexes add their own instability at the margin — two near-tied candidates can swap on different shards or after a partial index refresh.

Context you did not control. Location, interface language, logged-in state, memory or personalisation features, device, and prior turns in the conversation all condition the answer. A logged-in account with months of history is not sampling the same distribution as a clean browser session.

Time. Models get updated, retrieval indexes refresh, and providers run server-side experiments. A difference between Tuesday and Thursday can be a product change you will never be told about.

What does a single check actually tell you?

Treat "does my brand appear in the answer?" as a Bernoulli trial with unknown probability p. One run gives you a single 0 or 1. The maximum-likelihood estimate from n=1 is 0% or 100%, and the confidence interval spans essentially the whole range. You have learned that p is not exactly zero (if you appeared) or not exactly one (if you didn't). That is all.

This matters because the quantity you care about — how often a buyer asking this class of question sees you — is p. The individual answer text is a sample, not the population.

How many runs do you need?

The normal-approximation half-width for a proportion is roughly 1.96 × sqrt(p(1-p)/n). At the worst case of p = 0.5:

Runs (n)Approx. 95% margin of error
5±44 pp
10±31 pp
20±22 pp
30±18 pp
50±14 pp
100±10 pp

At small n the normal approximation is poor and you should use a Wilson interval instead. Seven mentions out of ten runs gives a Wilson 95% interval of roughly 40% to 89%. A dashboard that renders that as "70% visibility" with a trend arrow is reporting noise with high confidence.

from statsmodels.stats.proportion import proportion_confint
proportion_confint(7, 10, alpha=0.05, method="wilson")
# (0.397, 0.892)

The practical read: 10 runs per prompt tells you whether you are roughly never, sometimes, or usually cited. 30–50 runs gets you a number worth putting next to a competitor's. Fewer than 10 is a vibe check, which is fine as long as you label it as one.

Why repeating the same prompt isn't enough

Runs have to be independent in the ways that matter, or you are just measuring one draw several times.

How do you tell a real change from noise?

Comparing two proportions needs far more data than estimating one. A standard power calculation for detecting a 10-percentage-point difference around a baseline of 45%, at 80% power and α = 0.05, lands near 390 observations per group. If your before/after test is 30 runs against 30 runs, you can only detect enormous swings — roughly the difference between "rarely cited" and "usually cited."

Two ways to get leverage without a thousand API calls:

  1. Pool across prompts. You usually care about a topic, not a string. Aggregate runs across your prompt set and treat prompts as blocks, so between-prompt differences don't inflate your error term.
  2. Measure something continuous. Binary mention throws away information. Citation rank within the source list, number of sub-queries your domain was retrieved for, or whether the brand is named in prose versus only linked in a sidebar all carry more signal per run than a 0/1.

And hold a control: track two or three competitor domains in the same runs. If everyone's mention rate moved the same direction on the same day, you observed a platform change, not your content.

What to do

ai searchmeasurementstatisticsllm

WriteMySEO produces marketing content, not legal, medical, financial, or compliance advice. Figures cited reflect publicly reported industry data at time of writing and shift over time.

Get started

We write this well about your industry, every month.

AI-drafted, human-reviewed SEO content on a flat subscription. Blog posts, metadata, schema, and internal links, shipped on a monthly rhythm.

See plans

More from the blog