Reliability Diagrams and Calibration Curves

A model that predicts whether a trade will be profitable might output "0.85" for one setup. Does that mean setups like this one work out 85% of the time, or is 0.85 just a score the model happens to output for its most confident cases — really more like 60% in reality? Accuracy and AUC do not answer this. A model can rank cases correctly, and even classify most of them right, while its actual probability numbers are systematically wrong. If you size a position or set a stop using that number, the difference matters directly.

Calibration is the property of a probability being honest: among all the times the model said "70%," the event happened about 70% of the time. A reliability diagram is the plot that checks this directly.

Building the diagram

Take the model's predicted probabilities on a test set and sort them into bins — say, 0–10%, 10–20%, and so on. Within each bin, compute two numbers: the average predicted probability, and the actual fraction of positive outcomes among those cases. Plot predicted (x-axis) against actual (y-axis). A perfectly calibrated model produces points sitting exactly on the diagonal $y = x$. Points below the diagonal mean the model is overconfident in that range — it says 70% but only about 50% happen. Points above mean it is underconfident.

Worked example

A model makes 200 predictions that fall in the "0.80–0.90 predicted" bin, averaging 0.85. Of those 200, 140 actually turned out positive: an observed rate of $140/200 = 0.70$. That bin sits well below the diagonal — the model claims 85% confidence but earns 70%. If you had sized a bet proportional to the stated 0.85, you overstated your edge by 15 percentage points on every one of those 200 trades. Rescaling the model's outputs (with a technique like Platt scaling or isotonic regression) so the 0.85 bin actually maps to roughly 0.70 fixes the sizing without touching the model's ranking ability at all.

Why tree ensembles and neural nets need this

Models optimised purely to separate classes, like boosted trees or deep networks, have no reason to produce honest probabilities as a byproduct — they only need the ranking to be right to classify well. Boosted trees tend to push predictions toward 0 and 1 more aggressively than the true frequencies warrant; large neural networks, even when highly accurate, are frequently overconfident across the board. Both are calibrated after the fact, not by trusting the raw output.

A single number: the Brier score

A reliability diagram is a picture; sometimes you want one number to compare two models or track calibration over time. The Brier score is the mean squared difference between each predicted probability and the actual outcome (coded 0 or 1): $\frac{1}{N}\sum (p_i - y_i)^2$. It rewards both good calibration and good discrimination at once, so a falling Brier score does not by itself tell you which one improved — it is a summary to monitor alongside the diagram, not a replacement for it. Some practitioners decompose the Brier score into a calibration term and a resolution (discrimination) term specifically so the two effects can be separated; the arithmetic is more involved, but the reliability diagram already shows the same split visually — how close the curve sits to the diagonal is the calibration story, how far it spreads across the x-axis is the resolution story.

Fixing miscalibration

Two standard post-processing fixes exist once a reliability diagram shows a problem, and both work by learning a mapping from the model's raw output to a corrected probability using a held-out calibration set. Platt scaling fits a simple logistic curve through the raw scores, which works well when the miscalibration is a smooth, roughly S-shaped distortion — the common case for boosted trees. Isotonic regression fits a more flexible, monotonic step function, which can correct more irregular miscalibration at the cost of needing more calibration data to avoid overfitting the correction itself. Either way, the calibration set must be separate from both the training data and the final evaluation set, or the fix inherits the same leakage problem that plagues feature selection and stacking elsewhere in this curriculum.

Where quants meet this

Anywhere a probability feeds a decision that scales with its size — Kelly-style position sizing, expected-value trade filters, risk limits keyed to a "probability of breach" — miscalibration turns directly into mis-sizing, not just misclassification. This is also the standard diagnostic behind Calibrating a Forecast: Are Your Numbers Real?, and the fix for deep models specifically is covered in Calibrating Neural Network Probabilities. A model you only ever use to rank candidates (top decile vs. bottom decile) can tolerate poor calibration; a model you plug a number from into a formula cannot.