You ask ChatGPT a question and your brand gets cited. A colleague asks the same question twenty minutes later and it doesn't. Neither of you changed anything on the site.
That is not a bug in the tool or a sign that something broke. It is the expected behaviour of a stochastic system being sampled once. Most "AI visibility" reporting treats a single run as a measurement, when it is actually a single draw from a distribution nobody has estimated. Until you know the shape of that distribution, you cannot say whether your 38% mention rate went up, went down, or did nothing at all.
Where does the variance actually come from?
Four independent sources stack on top of each other, and only one of them is the model's decoding temperature.
Decoding randomness. Assistants sample tokens rather than taking the argmax. Different token choices early in the answer change which entities get named later. Even where a provider exposes a temperature of zero, exact reproducibility is not something hosted inference guarantees — batching, mixed precision, and expert routing in sparse models all make bit-identical outputs a non-goal. That is reasonable inference from how the systems are built, not a documented commitment either way.
Retrieval randomness. This is the bigger source. Google has described AI Mode as using a query fan-out technique: the system issues multiple related searches and synthesises across the results. The generated sub-queries are themselves model output, so they vary run to run. Different sub-queries hit different result sets, which means different candidate documents reach the generator. Approximate nearest-neighbour indexes add their own instability at the margin — two near-tied candidates can swap on different shards or after a partial index refresh.
Context you did not control. Location, interface language, logged-in state, memory or personalisation features, device, and prior turns in the conversation all condition the answer. A logged-in account with months of history is not sampling the same distribution as a clean browser session.
Time. Models get updated, retrieval indexes refresh, and providers run server-side experiments. A difference between Tuesday and Thursday can be a product change you will never be told about.
What does a single check actually tell you?
Treat "does my brand appear in the answer?" as a Bernoulli trial with unknown probability p. One run gives you a single 0 or 1. The maximum-likelihood estimate from n=1 is 0% or 100%, and the confidence interval spans essentially the whole range. You have learned that p is not exactly zero (if you appeared) or not exactly one (if you didn't). That is all.
This matters because the quantity you care about — how often a buyer asking this class of question sees you — is p. The individual answer text is a sample, not the population.
How many runs do you need?
The normal-approximation half-width for a proportion is roughly 1.96 × sqrt(p(1-p)/n). At the worst case of p = 0.5:
| Runs (n) | Approx. 95% margin of error |
|---|---|
| 5 | ±44 pp |
| 10 | ±31 pp |
| 20 | ±22 pp |
| 30 | ±18 pp |
| 50 | ±14 pp |
| 100 | ±10 pp |
At small n the normal approximation is poor and you should use a Wilson interval instead. Seven mentions out of ten runs gives a Wilson 95% interval of roughly 40% to 89%. A dashboard that renders that as "70% visibility" with a trend arrow is reporting noise with high confidence.
from statsmodels.stats.proportion import proportion_confint
proportion_confint(7, 10, alpha=0.05, method="wilson")
# (0.397, 0.892)
The practical read: 10 runs per prompt tells you whether you are roughly never, sometimes, or usually cited. 30–50 runs gets you a number worth putting next to a competitor's. Fewer than 10 is a vibe check, which is fine as long as you label it as one.
Why repeating the same prompt isn't enough
Runs have to be independent in the ways that matter, or you are just measuring one draw several times.
- Same conversation, same session. Later turns are conditioned on earlier ones. Start a fresh session every run.
- Same account. Personalisation and memory features make an account a fixed effect. Rotate accounts, or use logged-out/API access and state that's what you did.
- Same minute. Burst all 30 runs at once and you have sampled one index state. Spread runs across days and times of day, and treat the day as a blocking factor.
- Same phrasing. One prompt string is one query, not a topic. If you care about "best project management software for agencies," sample a set of 10–20 realistic phrasings and report across them. Phrasing variance is usually larger than run-to-run variance for the same string, and it is the variance your buyers actually generate.
How do you tell a real change from noise?
Comparing two proportions needs far more data than estimating one. A standard power calculation for detecting a 10-percentage-point difference around a baseline of 45%, at 80% power and α = 0.05, lands near 390 observations per group. If your before/after test is 30 runs against 30 runs, you can only detect enormous swings — roughly the difference between "rarely cited" and "usually cited."
Two ways to get leverage without a thousand API calls:
- Pool across prompts. You usually care about a topic, not a string. Aggregate runs across your prompt set and treat prompts as blocks, so between-prompt differences don't inflate your error term.
- Measure something continuous. Binary mention throws away information. Citation rank within the source list, number of sub-queries your domain was retrieved for, or whether the brand is named in prose versus only linked in a sidebar all carry more signal per run than a 0/1.
And hold a control: track two or three competitor domains in the same runs. If everyone's mention rate moved the same direction on the same day, you observed a platform change, not your content.
What to do
- Define the unit of measurement as a prompt set (10–20 phrasings), not a prompt.
- Run each phrasing at least 10 times, spread across at least three days, in fresh logged-out sessions with location and language pinned and recorded.
- Report a Wilson interval next to every percentage. If the interval crosses your previous interval, write "no detectable change."
- Log the full response, the cited URLs, and the citation order — not just a boolean. You will want to re-analyse later.
- Track competitors in the same runs as a control for platform-level shifts.
- Before running a before/after test, compute the detectable effect size for your sample budget. If you can only detect a 25-point swing, say so in the brief rather than discovering it at readout.