Calibration Curves and Reliability Diagrams

A model outputs "72% probability this trade is profitable." Is that number trustworthy, or is it just a score that happens to be squeezed between 0 and 1? Those are different questions. A model can rank opportunities perfectly — always giving higher scores to genuinely better trades — while its actual numbers are badly miscalibrated, calling everything "72%" when the true rate is closer to 50%. Ranking quality (AUC) and calibration quality answer different questions, and a lot of models are optimized for the first while quietly failing the second, which matters enormously the moment anyone tries to size a position using the raw probability.

What "calibrated" actually means

A model's predictions are calibrated if, among all the times it predicts a probability of $p$, the outcome actually happens close to a fraction $p$ of the time. Predict 30% on a thousand different events, and calibration means roughly 300 of them should actually occur — not 150, not 600. This is a statement about the average behavior of the model's outputs across many predictions, not a claim about any single prediction being "right."

A reliability diagram makes this checkable. Bin the model's predicted probabilities into groups — say 0–10%, 10–20%, and so on — and for each bin plot the average predicted probability on the x-axis against the actual observed frequency of the outcome in that bin on the y-axis. A perfectly calibrated model traces the diagonal line $y = x$. A model whose points sit below the diagonal is overconfident — it says "80%" when the true rate is closer to 60%. A model whose points sit above the diagonal is underconfident, saying "40%" for things that happen 60% of the time.

Worked example: building a reliability diagram by hand

A model scores 100 trade setups as "likely profitable," split into two probability bins. In the 60–70% bin, the model placed 40 predictions with an average predicted probability of 65%; of those 40, 22 turned out profitable — an observed rate of 22/40 = 55%. In the 80–90% bin, the model placed 30 predictions with an average predicted probability of 85%; of those 30, 27 turned out profitable — an observed rate of 27/30 = 90%.

Plotting these: the 65%-predicted bin sits at (0.65, 0.55) — below the diagonal, meaning the model is overconfident in that range. The 85%-predicted bin sits at (0.85, 0.90) — slightly above the diagonal, meaning the model is if anything a touch underconfident there. This kind of pattern, overconfident in the middle and roughly fine at the extreme, is common in models trained with a loss function that rewards sharp, decisive scores over well-calibrated ones.

What this means in practice

Calibration matters most exactly where a model's probability is used as an input to a further calculation, not just a ranking. Position sizing that scales bet size with predicted win probability (a Kelly-style rule, for instance) will systematically over-bet if the model is overconfident, because it treats an inflated "80%" as if it were genuinely 80%. Miscalibration can be corrected after the fact — Platt scaling and isotonic regression are the two standard fixes, both of which learn a mapping from the model's raw scores to better-calibrated probabilities using a held-out validation set — but that correction only works if calibration is checked in the first place, which it often isn't, because a model can look excellent on AUC while being unusable as an actual probability.

Practice

1. A weather-style model predicts "90% chance of a positive earnings surprise" for 50 stocks, and 30 of them actually surprise positively. Is the model overconfident, underconfident, or roughly calibrated in this bin?
2. Explain why a model with perfect AUC (it always ranks the eventual winner above the eventual loser) can still be badly miscalibrated.