You shipped a title tag rewrite across 4,000 product pages on a Tuesday. Clicks are up 9% two weeks later. Was it the change?
You cannot know from that comparison. In the same two weeks, seasonal demand moved, a competitor changed their pricing page, Google shipped something it did not announce, and your index coverage drifted. A before-and-after read on the treated pages measures your change plus everything else that happened to those pages. The whole point of an SEO experiment is to build a counterfactual: what would those pages have done if you had left them alone.
Why the unit of randomization is the URL, not the user
Conventional CRO testing randomizes users. You cannot do that for organic search, because the thing you are trying to move — how the ranking system evaluates a page — is a property of the URL, not the visitor. Serving different HTML to different users on the same URL also gets you nowhere: Googlebot sees one version, and whichever version it sees is the one that gets evaluated.
So SEO split tests randomize URLs. You take a pool of structurally similar pages, split them into a variant group and a control group, apply the template change only to the variant group, and serve that variant consistently to everyone including crawlers. There is no cloaking involved: each URL has exactly one version, and that version is what both users and Googlebot get.
The consequence is that your sample size is your page count, not your session count. A site with 40 category pages cannot run a meaningful split test on category templates. A site with 40,000 product pages can.
Why before-and-after comparison fails specifically
Pre/post analysis assumes the only thing that changed between the two windows is your intervention. In organic search, at least four things break that assumption:
- Seasonality and demand shifts. Impressions move with query volume you do not control. A rising tide lifts the treated pages and you take credit.
- Algorithm updates. Core updates land during multi-week test windows regularly. They do not affect all page types equally.
- Index churn. Pages enter and leave the index during the window. If your "after" period has more indexed pages in the pool, aggregate clicks rise for reasons unrelated to the change.
- Crawl lag. Google has to re-crawl and re-process the changed pages before any effect can exist. That lag varies by page depth and crawl demand, so your "after" period contains a mix of processed and unprocessed URLs.
A control group absorbs all four. If seasonality lifts the variant group, it lifts the control group too, and the difference between them is still attributable to the change.
How to build groups that are actually comparable
Random assignment across the whole pool is the naive approach and it is usually not good enough, because organic traffic is extremely skewed. A handful of pages carry most of the clicks, and a single unlucky draw puts three of them in one group.
Stratify first. Bucket the pool by baseline click volume (deciles work), and randomize within each bucket. Optionally stratify on a second dimension that predicts behavior — category, page age, template variant.
import hashlib
def assign(url, salt="titletest-2024-11"):
h = hashlib.md5((salt + url).encode()).hexdigest()
return "variant" if int(h[:8], 16) % 2 else "control"Hashing the URL with a per-test salt makes assignment deterministic and reproducible: your CDN, your CMS, and your analysis script can all recompute it without a shared lookup table. Store the assignment anyway, because you want an audit trail of which URLs were in which arm on which dates.
Then validate the split before you ship anything. Pull 8 to 12 weeks of daily clicks and impressions per URL from the Search Console API, aggregate by group, and plot them. If the two lines do not track each other closely in the pre-period, your groups are not comparable and no amount of statistics will fix that afterward.
Choosing a model: difference-in-differences or a synthetic control
Two reasonable analysis approaches, depending on how well your groups match.
Difference-in-differences is the simple one. Compute the change in the variant group from pre-period to post-period, compute the same for the control group, and subtract. It requires that both groups would have followed parallel trends absent the intervention — which is exactly what your pre-period plot is testing.
Bayesian structural time series is the better one when trends are not perfectly parallel. Google's open-source CausalImpact package fits a model of the variant group's traffic using the control group as a predictor over the pre-period, then projects that model forward and reports the gap between projection and reality, with a credible interval. It handles gradual trends and weekly seasonality that a two-point difference cannot.
Either way, report an interval, not a point estimate. "Clicks up 4.2%" is not a result. "Clicks up 4.2%, 95% interval −1.1% to 9.4%" tells you the honest answer is we can't tell yet.
How long to run and what you can realistically detect
There is no universal answer, and anyone quoting one is guessing. The detectable effect size depends on three things you can measure for your own pool:
- Day-to-day variance of group-level clicks in the pre-period. Higher variance, longer test.
- Group size. Detectable effect shrinks roughly with the square root of the number of URLs.
- Crawl and re-processing lag. Nothing before the change is processed counts as signal.
Handle the third directly: confirm the change is live in Google's rendered version of a sample of variant URLs, and check server logs for crawl hits on the variant group. Then discard the burn-in window from analysis rather than letting it dilute the effect. Two to four weeks of post-burn-in data is a common working range; run the pre-period numbers to check whether that is enough for your site.
What you cannot split test this way
| Change | Testable? | Why |
|---|---|---|
| Title/meta templates on a page class | Yes | Per-URL, plenty of units |
| Internal linking module in a template | Carefully | Links from variant pages leak PageRank into control pages |
| Schema markup additions | Yes | Per-URL, but measure eligibility/appearance, not just clicks |
| Sitewide navigation change | No | One unit, no control |
| Domain migration or hostname change | No | One unit |
| Robots.txt or canonical policy | Usually no | Site-level, and contamination is severe |
Contamination is the underrated failure mode. If your variant pages link differently, they change the control pages' link profile, which biases the comparison toward zero — you will underestimate a real effect. Where internal linking is the thing you are changing, segment by directory or category rather than randomizing URL-by-URL, and accept the weaker inference.
What to actually do
- Pick a page class with at least several hundred URLs and stable daily traffic.
- Pull 8–12 weeks of daily per-URL clicks and impressions from the Search Console API.
- Stratify by baseline click decile, hash-assign within strata, and plot both groups for the pre-period. If they do not track, restratify.
- Ship the change to the variant group only, then verify crawl and re-processing on a sample before starting the clock.
- Analyze with CausalImpact or difference-in-differences, report a credible interval, and pre-commit to your post-period length so you are not stopping the test the day it looks good.
Most SEO "wins" are unfalsifiable claims about a moving system. A control group is the cheapest thing you can add that turns them into evidence.