Reliability Diagrams for Trading Forecasts

A model that says "I'm 80% confident this trade goes up" is making two claims at once: a direction, and a reliability claim about that 80%. If trades the model tags as 80%-confident actually go up only 55% of the time, the direction call might still be useful, but sizing positions off that stated 80% will be badly wrong. A reliability diagram is the standard way to check the second claim — whether stated confidence matches reality — separately from checking whether the direction call itself is any good.

Building the diagram

Take every prediction a model made (say, "probability this stock closes up tomorrow"), bucket predictions into bins — 0-10%, 10-20%, and so on — and within each bin compute two numbers: the average predicted probability, and the fraction of times the outcome actually happened. Plotting bin-average-predicted against bin-actual-frequency, one point per bin, traces the reliability curve.

Move the probability parameter above and picture each of a model's confidence bins as one of these binomial outcomes: a model that predicts "70% probability" for a batch of trades should see roughly 70% of them succeed — the explorer shows how much a realized frequency can wobble around a true probability even when the model is perfectly calibrated, purely from sample size.

Worked example

A signal makes 200 predictions it tags as "65% confident, up." Of those 200, 110 actually closed up — a realized frequency of $110/200 = 55\%$. The point for this bin sits at (predicted 0.65, actual 0.55), ten percentage points below the diagonal: the model is overconfident in this bucket. Meanwhile its "50% confident" bucket, covering 300 predictions, saw exactly 152 go up, a frequency of $152/300 \approx 50.7\%$ — essentially on the diagonal, well calibrated.

The fix used in practice is not to throw out the 65% bucket's directional call, but to recalibrate it: since predictions tagged 65% actually resolve at about 55%, a trader sizing positions off this model should treat any future "65%" prediction as worth roughly 55% until the model itself is refit and retested.

What this means in practice

A model can have real, useful directional skill (it correctly separates likely-up from likely-down trades) while still being badly calibrated (its stated probabilities are systematically too extreme). These are different failures needing different fixes — better features address the first, recalibration (like fitting a simple correction curve on top of the raw output) addresses the second, and reliability diagrams are how a researcher tells which one they're facing.