You can split-test SEO on a fully static site with no server logic at all: assign similar pages to buckets at build time, apply the change to the treatment bucket only, and compare Search Console clicks with a difference-in-differences analysis. The honest prerequisites are the part nobody quotes: you need roughly 200+ pages per bucket, 1,000+ combined weekly clicks per bucket, and 4–6 weeks of patience per test. Below that, you're reading noise and calling it a result.
This is not user A/B testing
The unit of randomisation is the page, not the visitor. Every visitor — and Googlebot — sees the same version of any given page, so there's no cloaking risk; Google's own testing guidance only objects to showing crawlers something users don't get. What you're testing is whether a template change (title pattern, meta description, heading structure, adding an FAQ block, moving the comparison table up) moves clicks across a population of similar pages.
That framing also defines what you can't test this way: one-off pages, and changes whose effect is site-wide (navigation, internal linking architecture). Those only support pre/post analysis, which is much weaker.
Bucketing that doesn't lie to you
Random assignment over pages with wildly different traffic gives you buckets that differ by luck. Two fixes, both cheap:
Stratify. Sort pages by 90-day clicks, then alternate assignment down the ranked list so both buckets get the same traffic profile. Exclude the top 1% of pages entirely — a single head page swinging 15% on its own will dominate any effect you're trying to measure.
Make assignment deterministic. Hash the slug so rebuilds, added pages, and re-runs don't reshuffle buckets mid-test:
import hashlib
def bucket(slug: str) -> str:
h = int(hashlib.sha256(slug.encode()).hexdigest(), 16)
return "treatment" if h % 2 else "control"Combine both: stratify into traffic bands, hash-assign within each band. On this site we test within a page archetype only — comparison pages against comparison pages — because a title pattern that lifts "X vs Y" pages tells you nothing about category hubs.
Running it at build time
In any static generator this is one conditional in the template. Example: testing whether a year in the title lifts CTR:
control: {app} Alternatives: {n} Self-Hosted Options
treatment: {n} Best Self-Hosted {app} Alternatives in 2026Ship the change for the treatment bucket, record the deploy date, then update the sitemap lastmod honestly for changed pages so Google recrawls them. And here's the trap that invalidates half of all static-site SEO tests: the test starts when pages are recrawled, not when you deploy. Long-tail pages can take one to three weeks to be refetched. Track recrawl either from server logs or by sampling pages in the URL Inspection API, and start your measurement clock per-page from recrawl where you can, or globally from ~2 weeks post-deploy as a crude fallback.
Measurement: difference-in-differences
Single-bucket before/after is worthless in SEO because the background moves constantly — seasonality, core updates, competitors. The control bucket exists to absorb all of that. The estimate is:
effect = (treatment_after − treatment_before) − (control_after − control_before)Pull daily clicks per page from the GSC API (the UI truncates; the API gives you everything), aggregate per bucket, use a 4-week pre-period and 4–6 week post-period. For a defensible read, feed the same data to CausalImpact with the control bucket's series as the covariate — it produces a credible interval instead of a point estimate, which keeps you honest about whether the test actually resolved. Store the raw pulls; GSC only retains 16 months, and your self-hosted analytics won't reconstruct impressions after the fact.
Sample size reality
Rules of thumb from running these on directory-scale sites, assuming you're measuring clicks with typical week-to-week noise of 10–15% per bucket:
| Combined weekly clicks per bucket | Smallest lift you can reliably detect | Test duration |
|---|---|---|
| ~500 | ~25–30% | 6+ weeks |
| ~2,000 | ~10–15% | 4–6 weeks |
| ~10,000 | ~5% | 4 weeks |
| ~50,000 | ~2–3% | 3–4 weeks |
Most template changes produce single-digit lifts. Read the table pessimistically: with 500 clicks a week per bucket, only a dramatic win is even visible, so test dramatic changes (full title rewrites) rather than comma placement.
Pitfalls that end tests early
- A core update mid-test. If Google announces one, extend the test and lean on the control bucket; if the buckets diverge wildly during the rollout week, discard and re-run.
- Testing two things at once. One template change per test. A title and description changed together are unattributable.
- Peeking at day 5 and shipping the winner. Recrawl lag means early data is dominated by whichever bucket got refetched first.
- CTR mirages. A title that adds "2026" can lift CTR while attracting worse-fit impressions. Judge on clicks and post-click behaviour, not CTR alone.
What I'd test first
Title patterns, then result-snippet content (the first 160 characters of the page and the meta description), then above-the-fold structure — in that order, because that's the descending order of measured effect sizes I've seen. Titles routinely move clicks 10–30% on long-tail pages; meta descriptions move them a few percent when Google even uses yours. Set up the hash-bucketing once, keep a log of every test with dates and verdicts, and accept "no detectable effect" as the most common and most useful outcome — it's the finding that stops you cargo-culting changes across 50,000 pages.