SEO Split Tests Need a Control Group, Not a Before and After

Most reported SEO "wins" are a chart with a vertical line on it. Change shipped on the 12th, clicks went up after the 12th, therefore the change worked. That inference is only valid if nothing else about the world changed on the 12th — and in search, something always does.

Google ships ranking changes continuously. Demand moves weekly and seasonally. Your own site changes: new pages publish, old ones get pruned, internal links shift. A before/after comparison attributes all of that to your test. The fix is not a better chart. It is a control group that experiences everything except your change.

Why before/after comparisons fail mechanically

The quantity you actually want is a counterfactual: what would these pages have done if you had not touched them? You can never observe it, so you have to estimate it. A pre-period average is the laziest possible estimator, and it carries four specific biases:

• Demand drift. Query volume for your topics is non-stationary. Impressions can rise or fall without any ranking change at all. If you measure clicks, you have baked demand into your effect estimate.
• Algorithm updates. Core updates and smaller unannounced changes land during your measurement window. They hit your test pages and your untouched pages alike — which is exactly why an untouched group is useful.
• Regression to the mean. Teams usually test on pages that are underperforming right now. Underperformance is partly noise, and noise reverts. The page would have recovered somewhat anyway.
• Mix shift. If the test coincides with new URLs entering the template, or with cannibalizing siblings being consolidated, aggregate clicks move for reasons unrelated to the on-page change.

What you can actually randomize in SEO

In a conversion A/B test, the randomization unit is the user. In SEO it cannot be, because the "user" whose behavior you are trying to change is a ranking system that sees one version of each URL. Cloaking a variant to Googlebot is not a test design; it is a policy violation.

So the unit is the URL. You take a set of pages that share a template and a role — product detail pages, city landing pages, blog posts in one category — split them into test and control, and apply the change only to the test half. Both halves then live through the same updates, the same seasonality, the same site-wide work. This is the design SearchPilot (formerly Distilled ODN) has publicly described for server-side SEO testing, and the logic holds whether you buy a platform or do it with a templating flag.

For the split to work, the groups have to be comparable before the change. Random assignment gets you there in expectation, but with 200 URLs and a heavy-tailed traffic distribution, one group can easily inherit the single page that drives 30% of clicks. Two practical fixes:

1. Stratify before randomizing. Bucket pages by pre-period clicks (deciles work), then randomize within each bucket.
2. Check pre-period correlation. Plot daily clicks for both groups across 8+ weeks before the change. If the two series do not move together already, the control will not predict the test afterward, and no amount of modeling rescues it.

# per-page stratified assignment
pages.sort(key=lambda p: p.pre_clicks)
for i, p in enumerate(pages):
    p.group = "test" if (i // 1) % 2 == hash(p.url) % 2 else "control"
# then verify: sum(pre_clicks) per group within ~5%

When a split test is invalid because of interference

Randomized designs assume one unit's treatment does not affect another unit's outcome. SEO violates this more than most channels. Watch for three cases:

• Competing pages. If test and control pages compete for the same queries, improving one can demote the other. Your measured lift is then partly cannibalization, and it will not replicate when you roll out to everything.
• Internal link and crawl effects. Changes that alter internal linking or URL count change crawl distribution across the whole site, including the control group.
• Site-wide elements. You cannot split-test a navigation change, a CDN migration, or a robots.txt rule. There is no clean control. For those, use a time-series design with external controls and be explicit that the evidence is weaker.

If test and control pages rank for overlapping query sets, split by query cluster rather than by individual page: assign all pages in a topical cluster to the same arm.

Modeling the counterfactual

The simplest honest estimator is difference-in-differences on log-transformed metrics:

lift = (log(test_post) - log(test_pre)) - (log(control_post) - log(control_pre))

Logs matter because SEO effects are roughly multiplicative and traffic distributions are skewed. The control term subtracts out whatever happened to both groups.

A stronger option is Bayesian structural time series — the approach behind Google's open-source CausalImpact package. It fits a model of the test group's daily series using the control group as a predictor during the pre-period, then projects that model forward to generate a counterfactual with credible intervals. The headline output is cumulative effect over the post-period, with an interval that tells you whether zero is plausible. Its assumption is strict and worth stating: the control must not itself be affected by the intervention.

When does the clock start, and how long do you run?

The treatment does not begin when you deploy. It begins when Google crawls, renders, and reprocesses each page — and that is staggered across URLs by days or weeks. Measuring from deploy date blends treated and untreated pages in your test arm and dilutes any real effect toward zero.

Pull the last-crawl timestamp from the URL Inspection API, or from your server logs, for every test URL. Define the intervention date as the point where a large majority of the test group has been recrawled. Discard the transition window.

For duration, two constraints bind: you need enough post-period days for the model to separate signal from weekly seasonality (four full weeks is a reasonable floor), and you need enough volume. A group of pages with a handful of clicks per day will not produce a credible interval narrow enough to decide anything, no matter how long you wait. Check that first — estimate the pre-period daily variance and ask what lift size you could detect.

What to do next

1. Pick one template with at least a few dozen URLs and meaningful daily impressions. That is your testable surface; everything else needs a different method.
2. Export 10–12 weeks of daily per-URL data from the Search Console BigQuery export, not the UI.
3. Stratify by pre-period clicks, randomize within strata, and verify the two series track each other before you ship anything.
4. Ship to the test arm only. Record per-URL recrawl dates and set the intervention date from them.
5. Model with DiD first for a sanity check, then CausalImpact for an interval. Report the interval, not the point estimate.
6. Write down the interference risk before you start. If test and control pages compete for the same queries, say so in the result.