Megastudy
A massive field experiment that tests many interventions against the same outcome in one population at once.
What it means
A megastudy is a single large-scale randomized field experiment in which many distinct intervention ideas are tested simultaneously, each as an arm, on a common outcome within the same population and platform. By holding the setting, sample, and measure constant, it allows clean apples-to-apples comparison of dozens of approaches — something a scatter of separate small studies cannot deliver — and it guards against publication bias because all arms are reported together. Megastudies have repeatedly found that real-world effects are small and that experts poorly predict which interventions win, a humbling check on intuition. The limits are cost, the need for a large captive population, and uncertainty about whether a brief test transfers to other contexts. It matters because it raises the evidential bar for 'what works,' replacing one-off demonstrations with rigorous, comparable horse races at scale.
Examples
A gym megastudy tested 50-plus interventions to boost attendance across tens of thousands of members, finding most effects modest and several intuitive ideas no better than control.
Rather than run twenty small studies, a health system texts twenty different flu-shot reminders to random slices of the same patient list at once, then ranks them all on the same uptake measure.
A savings app randomly assigns thirty nudges — reminders, defaults, lotteries, framings — across its users simultaneously and reports every arm, so the ideas that flopped cannot quietly vanish from the record.
First described in Milkman, Duckworth and colleagues (2021).