Bayesian inference
Treat unknowns as probability distributions and update them with data.
What it means
Bayesian inference is an approach to statistics in which uncertainty about parameters is represented by probability distributions that are updated via Bayes' theorem as data accrue. Beginning from a prior distribution encoding existing knowledge, it combines this with the likelihood of the observed data to produce a posterior distribution that is the basis for all conclusions. Unlike frequentist methods, it yields direct probability statements about hypotheses and parameters, can incorporate prior information, and updates coherently as evidence arrives. Critics note that results depend on the choice of prior, while proponents argue that making assumptions explicit is a feature, not a bug.
Why it took two centuries
Bayes stated the rule in the 1760s, but for most models it was useless in practice. Turning a prior and a likelihood into a posterior means dividing by the marginal likelihood, an integral over every possible parameter value that has no closed form outside a few 'conjugate' cases like the Beta-Binomial. The method sat idle until Monte Carlo made the integral moot: instead of computing the posterior, you draw samples from it. The Metropolis algorithm and, decisively, Gelfand and Smith's 1990 use of the Gibbs sampler showed how; modern engines like Hamiltonian Monte Carlo, in Stan and PyMC, made it routine. Bayesian inference is common now largely because it finally became computable.
Reading a posterior
The output is a whole distribution over the unknown, and every summary is read from it. A point estimate is its mean, median or mode; uncertainty is a credible interval, the range holding, say, 95 percent of the posterior mass. Crucially, that interval means what people wrongly assume a confidence interval means: given the model and data, a 95 percent probability the parameter lies inside it. You can also read the posterior probability of a hypothesis as a number. Because the answer is a distribution, the uncertainty propagates cleanly into any decision or downstream model rather than collapsing to a single guess.
The trouble with priors
The prior is where the arguments live. With ample data the likelihood dominates and reasonable priors converge on the same posterior, so the choice barely matters. With little data, many parameters, or rare events, it matters a great deal, and the posterior inherits the prior's shape. Flat or 'non-informative' priors are not a neutral escape: they are not invariant to how you parameterize the model, so a prior that looks uninformative on one scale can be strongly informative on another. Careful practice treats the prior as a modeling choice to be checked, simulating from it before seeing data and testing whether conclusions survive reasonable alternatives.
Where it shows up
Bayesian inference is standard wherever evidence arrives in pieces or prior knowledge is real. Adaptive clinical trials update the estimated treatment effect as patients enroll, allowing early stopping for success or futility. Ecologists and epidemiologists estimate populations and infection rates from sparse, noisy counts; phylogenetics reconstructs evolutionary trees; astrophysics fits cosmological parameters. Navigation systems and robots run recursive Bayesian filters, the Kalman filter among them, to track position in real time. In machine learning it underlies Gaussian processes and Bayesian neural networks, which report calibrated uncertainty rather than a bare prediction.
Limits and caveats
Two cautions matter. First, every conclusion is conditional on the model: a poorly chosen likelihood or prior yields a confident but wrong posterior, with no automatic warning. Second, the two main uses are not equally robust. Estimating a parameter is fairly forgiving of the prior once data accumulate, but Bayes factors, the Bayesian tool for comparing hypotheses, stay acutely sensitive to prior width even in large samples, so a model-comparison verdict can hinge on a choice that estimation would shrug off. Computation adds risk of its own: MCMC can fail to converge, so diagnostics such as R-hat and checks for divergent transitions are not optional.
Examples
Starting with a broad belief about a coin's bias, each flip shifts the posterior toward the rate actually observed.
A satnav starts with a rough guess of your arrival time and narrows it at every junction you clear, ending with a range rather than one confident number.
A brewery estimating how many bottles per batch fail can begin from what similar lines usually show, then let each week's counts pull that distribution toward its own reality.
A clinician combines a disease's prevalence with a test's known accuracy, so a single positive result raises the probability of illness without settling it, especially for a rare condition.
Coast Guard search planners hold a probability map of a missing vessel's position and re-weight every grid square as each sweep returns empty, steering the next pass toward the likeliest water.
First described in Bayes (1763); Laplace; modern revival via Jeffreys, Lindley, and MCMC computation.
Key references
- van de Schoot, R., Depaoli, S., King, R., Kramer, B., Martens, K., Tadesse, M. G., Vannucci, M., Gelman, A., Veen, D., Willemsen, J., & Yau, C. (2021). Bayesian statistics and modelling. Nature Reviews Methods Primers, 1(1), 1-26. doi.org/10.1038/s43586-020-00001-2
- Gelman, A., Vehtari, A., Simpson, D., Margossian, C. C., Carpenter, B., Yao, Y., Kennedy, L., Gabry, J., Burkner, P.-C., & Modrak, M. (2020). Bayesian workflow. arXiv preprint arXiv:2011.01808. doi.org/10.48550/arXiv.2011.01808
- Kruschke, J. K., & Liddell, T. M. (2018). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review, 25(1), 178-206. doi.org/10.3758/s13423-016-1221-4
- Gelman, A., Simpson, D., & Betancourt, M. (2017). The prior can often only be understood in the context of the likelihood. Entropy, 19(10), 555. doi.org/10.3390/e19100555
- Gelfand, A. E., & Smith, A. F. M. (1990). Sampling-based approaches to calculating marginal densities. Journal of the American Statistical Association, 85(410), 398-409. doi.org/10.1080/01621459.1990.10476213
- Bayes, T., & Price, R. (1763). An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53, 370-418. doi.org/10.1098/rstl.1763.0053