SEO reporting has picked up the vocabulary of statistics without much of the practice. "Statistically significant improvement" appears in decks describing a comparison of two months of traffic, which is not a test of anything.
The underlying problem is worth understanding, because the fix is not more math — it is being clearer about what you can and cannot claim.
Why standard tests misbehave here
Observations are not independent. A t-test assumes each data point is an independent draw. Daily sessions are heavily autocorrelated: today looks like yesterday. Treating 90 days as 90 independent observations vastly overstates your effective sample size, which makes p-values far smaller than they should be.
There is no control group by default. Comparing this month to last month attributes every difference to your intervention, including seasonality, a competitor's launch, a core update, and the weather. Without a control, you are measuring the world.
The metrics are not normally distributed. Traffic per page is heavily right-skewed. Ranking positions are bounded and skewed. Tests assuming normality applied to these distributions produce confident nonsense.
Effect sizes are small and noise is large. Most legitimate SEO changes produce single-digit percentage effects against data whose week-to-week variance is larger than that.
What actually works
Difference-in-differences with a control group. Split comparable pages into treatment and control, apply the change to treatment only, and compare the change in each. The control absorbs the confounders. This is the single most valuable technique available and the most under-used.
Interrupted time series. For site-wide changes where no control group exists, model the pre-change trend and test whether post-change data departs from the projection. Requires a decent history and a stable trend, and it will not save you if a core update lands the same week.
Bayesian framing. Instead of "is this significant," ask "what is the probability the effect is positive, and how large is it likely to be." This maps far better onto how decisions actually get made, and it degrades gracefully with small samples rather than producing a binary verdict.
Effect sizes with intervals, always. "CTR rose 3.2 points, 95% CI [0.8, 5.6]" tells a reader everything. "The improvement was statistically significant" tells them nothing about magnitude, which is the part that determines whether to act.
Practical rules
Pre-register the analysis. Decide before you start: what you are measuring, how long you will run, what threshold triggers action. This is the cheapest defense against the strongest bias in the field, which is deciding what counts as success after seeing the data.
Use long enough windows. Four weeks minimum for anything CTR-related; longer for ranking effects. Shorter windows measure noise.
Aggregate to the right level. Individual URLs are noisy. Groups of similar URLs are much more stable. Test at the group level.
Account for multiple comparisons. Testing twenty things and reporting the one that reached p < 0.05 is a mathematical certainty, not a finding. If you are going to test many things, either correct for it or describe the results as exploratory.
Do not overclaim from a small effect. If the confidence interval includes zero, the honest statement is "we could not detect an effect," not "no effect" and certainly not "a small positive effect."
Talking to stakeholders
The instinct is to simplify uncertainty away, because "we think this worked, roughly this much, with this much confidence" sounds weaker than "this worked." It is weaker. It is also true, and the alternative has a failure mode: overclaimed results eventually get checked.
A framing that works in practice:
We changed X on 60 pages and held 60 comparable pages as a control. Over six weeks, the changed pages gained 8% more clicks than the control group. Given the variance in this data, the real effect is probably between 3% and 13%. We think it is worth rolling out, and we will re-measure at scale.
That is honest, actionable, and holds up if someone re-runs the numbers.
The larger point
Most SEO work does not need a formal test. Fixing a broken canonical, adding internal links to orphaned pages, or rewriting a title that truncates in the SERP are supported by mechanism and established practice. Demanding an experiment for each is a bad use of time.
Reserve testing for genuinely uncertain, genuinely reversible decisions where the answer changes what you do at scale. For everything else, act on the mechanism and measure the outcome — without dressing the measurement up as an experiment it was never designed to be.