Brier score
A scorecard for probabilistic forecasts that rewards being both accurate and well-calibrated.
What it means
The Brier score is a measure of the accuracy of probabilistic predictions, computed as the mean squared difference between the probabilities a forecaster assigned to outcomes and what actually happened, coded as one for events that occurred and zero for those that did not. Lower scores are better, ranging from zero for perfect forecasts to a worst case of one (or two, in some formulations) for confident wrong calls, and it penalizes overconfidence sharply because squaring magnifies large errors. It is a 'proper' scoring rule, meaning a forecaster minimizes their expected score only by reporting their true beliefs, so it gives no incentive to hedge or exaggerate. The score can be decomposed into components reflecting calibration (do events you call 80% likely happen about 80% of the time?), resolution (do your probabilities discriminate events that happen from those that don't?), and the base-rate uncertainty of the events themselves. It matters because it turns vague claims of being a 'good predictor' into a rigorous, comparable number, central to forecasting tournaments and the study of superforecasting.
Two scores share the name
Brier's 1950 paper defined the score across all forecast categories at once, summing the squared errors over every possible outcome, which for a two-outcome event can reach two. The version most people now call the Brier score is Murphy's later 'half-Brier': the mean squared error of the probability given to the single event that matters, bounded between zero and one. The two differ by a factor of two, so a number is only interpretable once you know which convention produced it. For several mutually exclusive categories the summed form generalizes naturally, and the familiar binary version is just its special case. When you compare forecasters or models, confirm both were scored the same way before reading anything into the gap between them.
Reading the number: the skill score
A raw Brier score is hard to judge on its own because its floor depends on how common the event is. Predicting rain in a desert scores well simply because 'dry' is almost always right, not because the forecaster is skilled. The Brier skill score fixes this by comparing the forecaster to a naive baseline, usually the long-run base rate (climatology): one means perfect, zero means no better than always guessing the base rate, and a negative value means worse than that baseline. This turns an absolute error into a relative verdict that travels across problems with different event frequencies. The same normalization lets clinical risk models and weather services report a performance figure that outsiders can actually compare.
Where it breaks down
The Brier score struggles with rare events. Because squaring makes small probabilities contribute little, a lazy model that always predicts the base rate can post a deceptively low score, and telling a genuinely skilful rare-event forecast apart from a useless one can demand thousands of observations. For heavily imbalanced problems many analysts prefer the logarithmic score, which punishes underconfident misses on rare outcomes more sharply. A subtler trap is treating a good Brier score as proof of calibration: the score blends calibration and discrimination into one number, so a well-discriminating but miscalibrated model can beat a well-calibrated but blunt one. If you care specifically whether '80%' means 80%, inspect the calibration component or a reliability diagram rather than the headline figure.
Brier versus the log score
Both the Brier and logarithmic scores are strictly proper, so both reward honest reporting, but they punish differently. The Brier score is quadratic and bounded: even a confidently wrong call costs at most one, so a single catastrophic miss cannot swamp an otherwise good record. The log score is unbounded and assigns an infinite penalty to a probability of zero placed on something that then happens, making it far less forgiving of overconfidence at the extremes. In fact the Brier score is a second-order approximation to the log score near well-calibrated forecasts, which is why the two usually rank forecasters similarly. Choose Brier when you want a stable, interpretable average; choose the log score when catastrophic overconfidence is the thing you most want to deter.
Examples
A weather forecaster who says 70% chance of rain on days it rains 70% of the time, and is appropriately uncertain otherwise, earns a low (good) Brier score; one who always blares 100% earns a high (bad) one.
Two analysts each get six of ten merger calls right. The one who said '60% likely' scores better than the one who blared '90% certain' — confidence you cannot back is punished.
Because the score rewards honesty, a doctor asked how likely a treatment is to work gains nothing by hedging at 50% to look safe; only their true belief minimizes the score.
An election model that assigns 85% to eventual winners across a hundred races is scored against a rival that calls every race a coin flip; the sharper, well-calibrated model earns the lower Brier score.
A spam filter outputs a probability, not a verdict. Tracking its Brier score over a month reveals whether its '90% spam' flags really are spam about nine times in ten, or whether it needs recalibrating.
First described in Glenn W. Brier (1950).
Key references
- Hoessly, L. (2025). On misconceptions about the Brier score in binary prediction models. arXiv preprint arXiv:2504.04906. doi.org/10.48550/arXiv.2504.04906
- Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., ... & Tetlock, P. E. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science, 25(5), 1106-1115. doi.org/10.1177/0956797614524255
- Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010). Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology, 21(1), 128-138. doi.org/10.1097/EDE.0b013e3181c30fb2
- Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378. doi.org/10.1198/016214506000001437
- Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12(4), 595-600. doi.org/10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2
- Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1-3. doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2