Experiment & Research Method
A/B Split Test
An A/B split test randomly shows two variants to comparable groups of users to measure which one moves a target metric.
- Fidelity
- High
- Effort
- Medium
- Time to run
- 1–4 weeks
What is a A/B Split Test?
An A/B split test (or split test) is a controlled experiment that randomly divides your audience into two groups: one sees the current experience (control, "A") and the other sees a single change (variant, "B"). Because assignment is random, any statistically significant difference in the target metric can be attributed to the change rather than to chance or confounding factors.
A/B testing is the workhorse of product experimentation because it produces causal evidence, not just correlation — the closest thing to a controlled lab experiment you can run on live traffic.
When to use it
Good fit
- You have enough traffic or conversions to reach statistical significance in a reasonable time.
- You want causal proof that a specific change moves a specific metric.
- The change is isolated enough to attribute the result cleanly.
Reach for something else when
- Traffic is too low to detect a meaningful effect before the result goes stale.
- You are exploring an open-ended problem — start with qualitative research instead.
- The change is so large or strategic that it needs a phased rollout rather than a 50/50 test.
How to run it
Form a falsifiable hypothesis
State the change, the expected effect, and the metric it should move: "Simplifying the signup form will increase signup conversion." A vague hypothesis produces an uninterpretable result.
Pick one primary metric and a minimum detectable effect
Choose the single metric that decides the test and the smallest lift worth shipping. These determine the sample size and runtime you need.
Calculate sample size and duration up front
Use a power calculation so you know before launching how long to run. Committing in advance prevents peeking and false positives.
Randomly split traffic and run without interference
Split users 50/50, change only the one variable, and let the test run its full duration — including complete business cycles (weekdays and weekends).
Analyze significance, then decide
Check whether the difference is statistically significant and practically meaningful. Ship the winner, iterate on a flat result, or roll back a loser — and record what you learned.
What you'll learn
A causal answer to whether a specific change improves a specific outcome, with a quantified effect size and confidence level — plus the guardrail metrics that tell you the change did not hurt anything else.
Frequently asked questions
How long should an A/B test run?
Long enough to reach the sample size your power calculation requires, and at least one to two full business cycles so weekday/weekend and pay-cycle effects average out. Ending a test the moment it looks significant ("peeking") is the most common cause of false wins.
What is the difference between A/B testing and multivariate testing?
An A/B test compares one change against a control. A multivariate test varies several elements at once to measure their individual and combined effects — which requires substantially more traffic to reach significance.
How much traffic do I need to run an A/B test?
It depends on your baseline conversion rate and the minimum effect you want to detect. Smaller effects and lower baseline rates need far more traffic. Run a sample-size calculation before launching; if the required runtime is many months, favor qualitative research first.