Calibrating a Forecast: Are Your Numbers Real?

A signal can rank stocks in exactly the right order — the highest-scored names really do outperform the lowest-scored ones — and still produce a forecast that is badly wrong in an absolute sense. If the model says "expect +8%" and the true expected return is actually +1.5%, position sizing built on that number will oversize every trade the model is confident about, even though the ranking the model produced was fine. This is a calibration problem, and it hides from the metrics researchers check most often: rank correlation and IC only ever ask whether the order is right, never whether the magnitude is right.

What calibration means, concretely

A forecast is calibrated if, among all the times it predicted some value $x$, the average realised outcome really was close to $x$. Bucket every forecast the model ever made into groups by predicted value, and compare each bucket's average prediction to that bucket's average realised return. A calibrated model's buckets sit on the 45-degree line — predicted equals realised, on average. A model can have a perfectly good IC and still miss this badly, if its outputs are systematically too extreme, too muted, or biased in one direction.

$\text{realised return} \approx m \cdot \text{predicted return} + b$

In words: regress realised returns on predicted returns across many forecasts. A calibrated model has slope $m$ close to 1 and intercept $b$ close to 0 — realised tracks predicted, one-for-one, with no offset. A slope well below 1 means the model is overconfident: its big forecasts don't pay off as big as promised. A slope above 1 means it is underconfident — its numbers understate how much the world actually moves in the direction it points. A nonzero intercept means the whole forecast is shifted, regardless of magnitude.

A worked example

A model's forecasts and the returns that followed, averaged into five buckets by predicted value:

Predicted return	Realised return (avg.)
+6%	+2.1%
+3%	+1.3%
0%	+0.1%
−3%	−1.0%
−6%	−1.9%

Regressing realised on predicted gives a slope of roughly 0.33 and an intercept near 0. The model is directionally right every time — positive predictions realise positive, negative realise negative, in the right relative order — but it is dramatically overconfident: realised returns come in at roughly a third of predicted magnitude, consistently, across the whole range. A position-sizing step trusting the raw +6% forecast is sizing for three times the return actually likely to show up.

This is exactly what a calibration check looks like in practice: plot predicted against realised, fit a line, and read the slope. Drag points to see how a handful of over-optimistic large forecasts can drag the fitted slope well below 1 even while every point sits on the correct side of zero.

The fix in this example is direct — since the miscalibration is a near-constant proportional overconfidence, multiplying every raw forecast by roughly 0.33 (which is exactly a shrinkage step) restores calibration without touching the model that produced the ranking in the first place.

Where it bites

Position sizing, risk budgeting, and any process that compares this signal's conviction to another signal's conviction all silently assume calibration. A miscalibrated but well-ranked model still shows a respectable back-tested Sharpe, because Sharpe cares about direction and proportion within the book, not absolute magnitude — which is exactly why calibration bugs survive so long: the metric everyone watches doesn't catch them.