Brier Score and Proper Scoring Rules

A model says there's a 70% chance a stock closes up tomorrow. It closes down. Was the forecast wrong? Not necessarily — a well-calibrated 70% forecast should still be wrong about 30% of the time, so a single miss tells you nothing. Judging probability forecasts requires a metric that scores the whole distribution of forecasts against outcomes over many trials, and it needs to be constructed so that a forecaster can't improve their score by lying about their true belief. That second requirement — being a "proper" scoring rule — is easy to get wrong, and the Brier score is the standard example that gets it right.

An analogy: betting your own money on your belief

Imagine you're forced to bet an amount proportional to your stated confidence, and your payout is calibrated so that your best strategy, mathematically, is to state your true belief exactly — betting more when you're truly more confident wins you more on average, and any attempt to shade your stated number up or down loses you money in expectation. A proper scoring rule is built the same way for a forecasting model: the rule is designed so a forecaster's expected score is optimized by reporting their genuine probability, never by rounding it toward 0 or 1 to look more decisive, and never by hedging toward 50% to avoid being penalized for a bold call that didn't pan out.

The formula, one piece at a time

For a single forecast, let $p$ be the predicted probability of the event happening, and let $y$ be 1 if it happened and 0 if it didn't. The Brier score for that one forecast is:

$$BS = (p - y)^2 .$$

In plain English: it's the squared distance between what you predicted and what actually happened, so a forecast of $p=0.9$ for an event that occurs ($y=1$) scores $(0.9-1)^2 = 0.01$ — very good — while the same forecast for an event that doesn't occur scores $(0.9-0)^2 = 0.81$ — a heavy penalty for confident misses. Average this across many forecasts to get the overall Brier score; lower is better, with 0 being perfect and 0.25 being what a forecaster who always says "50%" achieves on a 50/50 coin flip.

Why it's proper: the squared-error form means a forecaster's expected score is minimized only by stating their true probability. Suppose your true belief is $p^* = 0.7$. If you instead report $p=1.0$ to "look confident," your expected score becomes $0.7 \times 0 + 0.3 \times 1 = 0.30$, worse than reporting $0.7$ honestly, which gives $0.7 \times 0.09 + 0.3 \times 0.49 = 0.21$. The math itself discourages exaggeration.

Worked example: two forecasters over 4 trades

Forecaster A predicts probabilities of $0.9, 0.6, 0.8, 0.3$ for four trades that actually resolved $1, 1, 0, 0$. Their squared errors are $(0.9-1)^2=0.01$, $(0.6-1)^2=0.16$, $(0.8-0)^2=0.64$, $(0.3-0)^2=0.09$, averaging to:

$$BS_A = \frac{0.01+0.16+0.64+0.09}{4} = 0.225 .$$

Forecaster B, more cautious, predicts $0.6, 0.6, 0.5, 0.5$ for the same four trades: squared errors $0.16, 0.16, 0.25, 0.25$, averaging to $BS_B = 0.205$. Even though Forecaster A "called" two of the four outcomes with higher confidence, Forecaster B's more moderate, better-calibrated forecasts score better overall — Forecaster A's one badly wrong confident call (0.8 on a trade that failed) cost more than the modest gains from their two confident correct calls were worth.

What this means in practice

Brier score is the right tool for evaluating a probabilistic model — a signal generator that outputs a confidence rather than a hard buy/sell call — because it scores the whole forecast, not just whether the most likely outcome happened to occur. It's especially useful for comparing models that have been tuned to sound confident against models that report honest, well-calibrated probabilities: a model gamed to output extreme probabilities to look decisive will score worse whenever it's confidently wrong, which a plain accuracy metric wouldn't catch. Log loss is a common alternative proper scoring rule; both share the key property that honesty is the optimal strategy.