False discovery rate
The expected share of your 'significant' findings that are actually false.
What it means
The false discovery rate is the expected proportion of rejected null hypotheses that are in fact true — that is, the fraction of 'discoveries' that are false alarms. Controlling the FDR, rather than the stricter family-wise error rate, tolerates a few false positives in exchange for greater power to detect real effects, which suits settings with thousands of simultaneous tests. The Benjamini-Hochberg procedure ranks p-values and applies a graduated threshold to hold the FDR at a chosen level. It has become the default in genomics, neuroimaging, and other high-dimensional fields where Bonferroni would be crippling.
How the procedure works
The Benjamini-Hochberg rule turns a pile of p-values into a decision by grading them on a sliding scale. Sort the m p-values from smallest to largest. The smallest is judged against the strictest bar, essentially Bonferroni, and each one after it against a threshold that relaxes in proportion to its rank: the k-th smallest passes if it falls below k over m times the target rate. Find the largest rank that still clears its bar and reject everything up to it. The graduated ladder is what buys power. A lone strong finding must survive a harsh cutoff, but once several small p-values cluster near the bottom of the list they reinforce one another and the effective threshold loosens.
What it controls, and what it does not
The false discovery rate is an average, not a promise about your particular list. A rate of 5 percent describes the expected fraction of false positives across hypothetical repetitions of the whole analysis; in any single run the realized proportion of false discoveries can land well above or below it. The procedure also says nothing about which flagged items are the false ones, and it does not bound the chance that any specific finding is spurious. Independence, or a benign form of positive correlation, is assumed. Under strong or negative dependence the guarantee can slip, which is why Benjamini and Yekutieli later derived a more conservative variant that holds under any dependence structure, though it sacrifices power to do so.
Adaptive versions and the q-value
Plain Benjamini-Hochberg quietly assumes every null is true, so it spends its error budget as if none of the tests would ever turn up a real effect. When a fair share of effects are genuine, common in a well-designed screen, that assumption wastes power. Storey's refinement estimates the proportion of true nulls from the shape of the p-value distribution and rescales the threshold to reclaim it. It also reframes the output as the q-value: the false discovery rate analogue of a p-value, read as the smallest rate at which a given feature would be called significant. Reporting q-values lets readers pick their own tolerance after the fact rather than committing to one cutoff before seeing the data.
Why it took over, and where it bites
The false discovery rate displaced Bonferroni in genomics, neuroimaging, and large-scale experimentation because those fields test thousands of hypotheses at once, where controlling the family-wise error rate would reject almost nothing. But the method's answer depends on the company a test keeps. Because the threshold is computed from the whole batch, the same p-value can be significant in one analysis and not in another purely because of which other tests were included alongside it. Splitting one family into several, or padding a batch with hopeless tests, changes which findings survive. Deciding what belongs in the family, and fixing that before looking, is as consequential as the rate you choose.
Examples
Screening 20,000 genes for association with a disease, an FDR of 10% means roughly one in ten flagged genes is a fluke.
A brain scan tests tens of thousands of voxels at once. Uncorrected, activation turns up in a dead salmon, the demonstration that made multiple-comparison control impossible to shrug off.
A growth team runs fifty experiments a quarter and ships every winner. Holding the false discovery rate at 5% means accepting that about one shipped win in twenty is noise.
A quant fund backtests three hundred trading signals. Holding the false discovery rate at 10 percent concedes that roughly one signal in ten clearing the bar is a backtest artifact, not a real edge.
A plant monitors hundreds of sensors for drift each shift. Bounding the false discovery rate, rather than alarming on every p below 0.05, keeps engineers from chasing false alarms while still catching genuine faults.
First described in Benjamini & Hochberg (1995).
Key references
- Storey, J. D., & Tibshirani, R. (2003). Statistical significance for genomewide studies. Proceedings of the National Academy of Sciences, 100(16), 9440-9445. doi.org/10.1073/pnas.1530509100
- Genovese, C. R., Lazar, N. A., & Nichols, T. (2002). Thresholding of statistical maps in functional neuroimaging using the false discovery rate. NeuroImage, 15(4), 870-878. doi.org/10.1006/nimg.2001.1037
- Storey, J. D. (2002). A direct approach to false discovery rates. Journal of the Royal Statistical Society: Series B, 64(3), 479-498. doi.org/10.1111/1467-9868.00346
- Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29(4), 1165-1188. doi.org/10.1214/aos/1013699998
- Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57(1), 289-300. doi.org/10.1111/j.2517-6161.1995.tb02031.x