Conformal Prediction

Most models that report a confidence interval are trusting an assumption — that residuals are normally distributed, that the model is well-specified, that the training and test data come from the same distribution in a way the model's internal math correctly captures. Break any of those assumptions, which financial data does constantly, and the reported interval can be badly wrong while still looking authoritative. Conformal prediction sidesteps the assumptions almost entirely: it produces intervals with a guaranteed coverage rate using nothing but a held-out calibration set and the model's own errors on it, regardless of what the model is or how it was trained.

The analogy: a tailor who measures the last hundred customers

A tailor selling suits off the rack wants to know how big a size range to stock. Instead of trusting a theoretical formula for human body proportions, she simply measures how far off her "standard" sizing was for the last hundred customers who walked in — the actual gaps between predicted and true measurements. If 90 of those 100 customers fit within a half-inch of the standard prediction, she can promise, for the next customer, "your true size will be within half an inch of our estimate, with about 90% confidence" — a guarantee built entirely from observed track record, with no assumption about the shape of the human population.

The mechanism: calibrate on errors, not on the model

Split data into a training set and a separate calibration set the model never trained on. For each calibration example, compute a nonconformity score — typically just the absolute residual, $s_i = |y_i - \hat{y}_i|$, though more refined scores exist. Sort these scores and take the $\lceil (n+1)(1-\alpha) \rceil / n$-th quantile, call it $q$. For a new input, the prediction interval is:

$$\hat{y}_{\text{new}} \pm q$$

In words: find the error size that covered $(1-\alpha)$ of the calibration examples, and apply that exact same margin to every new prediction. The guarantee — under the one assumption that calibration and future data are exchangeable, meaning drawn in a way that doesn't systematically differ in order — is that the interval covers the true value at least $(1-\alpha)$ of the time, on average over new examples, no matter what model produced $\hat{y}$ or how badly its internal assumptions about the data are violated.

Worked example: building an interval by hand

A model predicts next-day returns; on a calibration set of 9 examples, the absolute residuals (in percent) are: $0.3, 0.5, 0.8, 0.4, 1.2, 0.6, 0.9, 0.7, 2.1$. Sorted: $0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.2, 2.1$.

For 90% coverage ($\alpha = 0.1$) with $n=9$, take the $\lceil (9+1)(0.9) \rceil = \lceil 9 \rceil = 9$-th smallest value, which is the largest one: $q = 2.1$. That single outlier residual sets the whole margin — with only 9 calibration points, the interval is forced to be conservative because one bad miss has to be accounted for. Today's prediction is $\hat{y} = 0.5\%$, so the conformal interval is $0.5\% \pm 2.1\%$, i.e. $-1.6\%$ to $2.6\%$. It's wide, but it's honest: it reflects the actual worst-case error the model made on data it hadn't seen, rather than a theoretical assumption that might understate the tail.

Compare a naive interval built from assuming residuals are normal with the calibration set's standard deviation ($\approx 0.56$): $0.5\% \pm 1.96(0.56\%) = 0.5\% \pm 1.10\%$, noticeably narrower. That narrower interval would have missed the $2.1\%$ outlier residual entirely — the normal-distribution assumption underestimated the tail risk that the raw calibration data actually contained.

What this means in practice

Conformal prediction wraps around any model — a linear regression, a gradient-boosted tree, a neural network — without needing access to its internals, which makes it attractive as a drop-in risk layer on top of models already in production. The catch is the exchangeability assumption: it holds reasonably well for i.i.d.-ish data, but financial time series are serially dependent and non-stationary, so a plain conformal interval calibrated on last year's data can under-cover during a regime shift. Time-series-adapted variants exist — using rolling or weighted calibration windows that upweight recent residuals — precisely to keep the guarantee meaningful when the world is drifting under the model, which it usually is (see Concept Drift Detection). Even with that caveat, conformal intervals are valuable specifically because they make no claim about why the model is uncertain, only an empirically grounded claim about how often it's been wrong by how much.

One more subtlety worth carrying forward: coverage is a statement about averages over many predictions, not a guarantee for any single one. A 90% interval will be wrong roughly 10% of the time by construction, and there's no way to know in advance which predictions fall in that unlucky tenth. That's a feature, not a flaw — it's the same honesty a well-calibrated weather forecast has when it says "70% chance of rain" and it doesn't rain: the forecast wasn't wrong, the outcome was simply the less likely branch of a distribution it described correctly.