Multiverse analysis
Also known as: Specification curve analysis
Run every reasonable version of an analysis and report the whole spread.
What it means
A multiverse analysis examines how a conclusion holds up across the full set of defensible choices in processing and analyzing data, rather than reporting a single path. By computing the result under every reasonable combination of exclusions, codings, covariates, and model specifications, it reveals whether a finding is robust or an artifact of arbitrary decisions. The display of results across specifications, sometimes called a specification curve, exposes how much researcher degrees of freedom could be driving an effect. It promotes transparency and humility, though it does not by itself separate which specifications are theoretically appropriate from which are merely possible.
Examples
Reporting an effect across all 1,000 plausible analytic pipelines, showing it is significant in 95% of them rather than cherry-picking one.
Before announcing that the new onboarding flow lifts retention, the analyst reruns it under every defensible definition of 'retained' and every sensible set of controls, then shows the whole spread.
A wellbeing result survives when outliers are trimmed at three standard deviations and disappears when trimmed at two. Running both, and reporting both, is the honest write-up.
First described in Steegen, Tuerlinckx, Gelman & Vanpaemel (2016); Simonsohn et al. specification curve.