Publication bias
Also known as: File-drawer problem
Positive, novel results get published while null results are quietly buried.
What it means
Publication bias, also called the file-drawer problem, is the distortion of the scientific record that arises because studies with statistically significant, novel, or 'positive' results are far more likely to be published than studies that find null or negative results. The mechanism operates at several gates: authors are less motivated to write up null findings, reviewers and editors favor surprising and significant ones, and journals prize novelty, so non-significant and replication studies languish unpublished in researchers' file drawers. The consequence is that the published literature systematically overstates both the existence and the magnitude of effects, because the visible studies are a biased, success-skewed sample of all the studies actually conducted. This is especially corrosive for meta-analyses, which pool published results and can therefore inherit and amplify the bias, though techniques like funnel plots attempt to detect it. The recognized remedies include study registries, registered reports that are accepted before results are known, and norms encouraging the publication of nulls. It matters because decision-makers and other scientists relying on the literature will believe effects are larger and more reliable than they truly are.
Examples
Ten studies on a treatment find nothing and stay in their drawers; the one that happens to reach significance by chance gets into a journal and is cited as evidence.
A supplement looks miraculous in the press because the trials finding nothing were never written up, while the two flattering ones reached journals and became the story everyone repeats.
A marketing team only presents its winning A/B tests to the board. Two years on, the 'proven playbook' is a highlight reel, and the tactics quietly stop working.
First described in Rosenthal (1979); the file-drawer problem.