CRPS for Probabilistic Forecast Evaluation

A model that forecasts "$102, plus or minus $1" and one that forecasts "$102, plus or minus $20" can both have the exact same point forecast, and if the outcome lands at $102 both look perfect by an error metric that only checks the central number. But they made very different claims, and only one of them should get full credit for being right — the tight one, if it's genuinely that confident and correct, is more useful; if it's overconfident and the outcome later lands at $130, it should be punished much harder than the wide forecast would be. Ordinary error metrics like mean absolute error only look at the single predicted number and can't tell these two forecasts apart. The Continuous Ranked Probability Score (CRPS) is built specifically to score the entire predicted distribution, not just its center.

What CRPS actually measures

CRPS compares the model's full predicted cumulative distribution — the forecast's entire curve of "probability the outcome is below this value," for every possible value — against the outcome that actually happened, represented as a step function that jumps from 0 to 1 exactly at the realized value. CRPS is the area between those two curves:

$$CRPS = \int_{-\infty}^{\infty} \big(F(x) - \mathbb{1}\{x \geq y\}\big)^2 \, dx ,$$

where $F(x)$ is the model's predicted probability that the outcome is at most $x$, $y$ is the actual outcome, and $\mathbb{1}\{x \geq y\}$ is 1 for every $x$ at or above the true value and 0 below it. In plain English: CRPS measures how far, on average, the model's entire predicted distribution sits from a perfect forecast that puts 100% certainty exactly on the true outcome — squaring and integrating so that being wrong across a wide spread costs more than being wrong by a little.

A useful property: if the predicted distribution collapses to a single point, CRPS reduces exactly to the absolute error between that point and the outcome — a strict generalization of a familiar point-forecast metric, not a wholly different idea.

Worked example: two forecasts, one outcome

Suppose a model predicts tomorrow's return is normally distributed with mean 0% and standard deviation 1%, while a second model predicts mean 0% with standard deviation 5%. The actual outcome is a return of 0.5%. Both models had the same central forecast (0%), so mean absolute error alone would rank the two models identically on a point-forecast basis for this one day. But CRPS integrates the squared gap between each model's entire predicted distribution and the step function at 0.5% — the tight (1% sd) distribution sits much closer to a step-function jump at a value only half a standard deviation away, so it scores a lower (better) CRPS than the wide (5% sd) distribution, whose probability mass is spread thin across a much larger range and sits further, on average, from the sharp step at the true outcome.

Why the shape of the distribution matters

If instead the outcome had been a 4% return — four standard deviations out for the tight model — CRPS would swing hard the other way: the tight model's near-total confidence that returns stay within ±1% is badly violated, and it's penalized heavily, while the wide model's 5% standard deviation made a 4% move a fairly unremarkable, well-covered possibility. CRPS rewards forecasts whose stated uncertainty matches reality — neither systematically too confident nor needlessly too vague — which is exactly the property a risk manager needs from a forecast used to size positions or set stop levels.

Explore how sample means from different parent distributions cluster — CRPS is scoring something conceptually similar: how well a model's whole predicted spread lines up with where outcomes actually land, not just whether the average guess was close.

What this means in practice

CRPS is the standard evaluation metric for probabilistic forecasters (DeepAR-style models, ensembles that output a distribution rather than a single number) precisely because it can't be gamed by simply narrowing your uncertainty bands to look confident — narrow bands only help CRPS if the outcomes genuinely land inside them consistently.