A/B testing
An online randomized experiment comparing two versions to see which performs better.
What it means
A/B testing is a controlled experiment in which users are randomly assigned to see one of two or more variants of a design, message, or feature, and outcomes are compared to determine which performs better. It is essentially a randomized controlled trial applied at scale to digital products, and randomization is what gives it its power: by allocating users to conditions by chance, it neutralizes confounds so that any reliable difference in outcomes can be attributed to the variant itself rather than to who saw it. It has become the workhorse of applied behavioral science in technology, marketing, and product development because it can run continuously on large numbers of real users at low cost. Its pitfalls, however, are well known and matter as much as its strengths: peeking at results and stopping as soon as significance appears inflates false positives, running many simultaneous tests creates a multiple-comparisons problem, statistically significant differences can be practically trivial, and short tests can miss novelty effects or longer-term consequences. It matters as the dominant method for evidence-based optimization, provided it is run with proper sample-size planning, pre-specified metrics, and attention to practical as well as statistical significance.
What the evidence shows
The headline finding from large-scale experimentation is how often good ideas fail. Only about a third of the ideas tested on Microsoft's experimentation platform improved the metric they were designed to improve. Manzi separately reports that of roughly 12,000 experiments Google ran in 2009, about a tenth led to a business change — a different measure, but the same direction. That failure rate argues for testing rather than against it: teams cannot sort winners from losers by inspection, which is why the method earns its keep. On mature surfaces genuine effects are usually fractions of a percent, so tests must be powered for small differences.
Why peeking breaks the math
A fixed-horizon p-value assumes you look once, at a sample size chosen in advance. Peeking violates that assumption: as data accumulates the test statistic wanders, and a wandering statistic will eventually cross any fixed threshold by chance alone. Stopping the moment it does turns a nominal 5% error rate into something much larger. Johari and colleagues show that at 10,000 samples, continuous monitoring can inflate Type I error five- to tenfold. The remedy is not willpower. Sequential and always-valid methods widen the threshold to pay for repeated looks, letting teams monitor continuously and stop when they choose while keeping error control intact.
Trust the experiment before the result
Before asking whether an effect is real, ask whether the experiment ran correctly. The workhorse diagnostic is the sample ratio mismatch: if a 50/50 split delivers 50.2/49.8 across millions of users, the deviation is far too large to be chance, and it signals that assignment, logging or redirection is broken. Fabijan and colleagues found roughly 6% of Microsoft experiments exhibit one, and an affected result should be discarded, because whatever corrupted the split has probably corrupted the metric. A/A tests, running a variant against itself, calibrate the platform by confirming it produces significant results only at the rate chance predicts.
Short tests, long-run effects
A two-week test measures a two-week response, which is not always the response that lasts. Novelty effects flatter changes that are merely new; primacy effects punish changes that disrupt habits. More subtly, users learn. Hohnhold and colleagues at Google found that raising ad load lifts revenue immediately while making users progressively less likely to engage with ads at all, an effect they call ads blindness, so a short test badly overstates the change: a 50% cut to mobile ad load looked sharply revenue-negative in the short run and came out long-run neutral or better. The countermeasure is a long-running holdout keeping a slice of users on the old experience for months, letting slow effects surface.
Using it in practice
The discipline is mostly front-loaded, and the hard part is the metric itself. An overall evaluation criterion trades engagement against revenue and guards against known harms, forcing a team to state in advance what it is willing to give up to win elsewhere; sample-size planning and a minimum shippable lift follow from it. Variance reduction buys real time: CUPED uses pre-experiment behavior as a covariate, cutting variance and shortening runtimes without weakening validity. And treat a spectacular win with suspicion. Twyman's law holds that any figure that looks interesting or different is usually wrong.
Examples
Showing 50% of visitors a green button and 50% a red one to compare click-through.
A charity shows half its visitors a headline asking for three pounds a month and half a lump-sum ask; because the split is random, the gap in donations cannot be blamed on who turned up.
A product team declares a winner on day three, the moment the result first looks significant. Run to the sample size they planned for, the lift quietly disappears.
A retailer tests a single checkout redesign but reads out twenty metrics, and one clears the 5% threshold: returning customers in one region spend more. Twenty independent looks at a change that does nothing would be expected to throw up about one such result anyway, so it is a hypothesis for a fresh test, not a finding.
A search team's ranking change holds a 3% click lift for a full two-week test. A holdout kept on the old ranking for six months shows the gap gone by month three: users had learned the new layout, then drifted back.
First described in Industrialized by web companies; rooted in RCT methodology.
Key references
- Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B testing intuition busters: Common misunderstandings in online controlled experiments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 3168-3177. doi.org/10.1145/3534678.3539160
- Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2022). Always valid inference: Continuous monitoring of A/B tests. Operations Research, 70(3), 1806-1821. doi.org/10.1287/opre.2021.2135
- Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi.org/10.1017/9781108653985
- Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., & Dmitriev, P. (2019). Diagnosing sample ratio mismatch in online controlled experiments: A taxonomy and rules of thumb for practitioners. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2156-2164. doi.org/10.1145/3292500.3330722
- Hohnhold, H., O'Brien, D., & Tang, D. (2015). Focusing on the long-term: It's good for users and business. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1849-1858. doi.org/10.1145/2783258.2788583
- Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140-181. doi.org/10.1007/s10618-008-0114-1
- Manzi, J. (2012). Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society. Basic Books. www.hachettebookgroup.com/titles/jim-manzi/uncontrolled/9780465023240/