Error Analysis and Residual Inspection

Two models can have the exact same RMSE and be completely different in every way that matters for deploying them. One might be off by a little on every single prediction, evenly, harmlessly. The other might be perfect 95% of the time and catastrophically wrong the other 5% — say, on every high-volatility day, or every earnings announcement — with those rare disasters averaging out to the identical RMSE. A single summary metric cannot distinguish these two situations. Only looking at the actual errors, one by one and grouped by context, can — that practice is error analysis, and its core tool is the residual, the gap between what a model predicted and what actually happened.

The analogy: a doctor reading the chart, not just the temperature

A patient's average temperature over a week might read a perfectly normal 98.6°F. But if that average is hiding a pattern — spiking to 103°F every evening and dropping to 96°F every morning — the average has erased the exact information a doctor needs. A responsible doctor doesn't stop at the weekly average; they look at the full trace, hour by hour, hunting for when and why it deviates. Error analysis is that same discipline applied to a model: don't stop at the one number, look at the individual residuals and ask when and why they're large.

The formal picture

The residual for example $i$ is simply $e_i = y_i - \hat{y}_i$, the true value minus the model's prediction. RMSE and similar metrics are aggregates: $\text{RMSE} = \sqrt{\frac{1}{n}\sum_i e_i^2}$ — in words, take every residual, square it (so positive and negative errors don't cancel and big misses count extra), average, and take the square root back to the original units. Error analysis instead studies the distribution of the $\{e_i\}$ directly: are they centered at zero (unbiased) or systematically shifted (biased)? Do they grow with the size of $y_i$ (heteroscedastic)? Are they larger for a particular subgroup — a sector, a time-of-day, a volatility regime — than others (a pattern the aggregate metric averages away)?

Worked example 1: two models, identical RMSE, different residual patterns

Model A's residuals on 6 test days are $\{1, -1, 1, -1, 1, -1\}$ (small, evenly split errors). Model B's residuals on the same 6 days are $\{0, 0, 0, 0, 0, -3.46\}$ (perfect five days, one large miss). Compute RMSE for each: Model A, $\sqrt{(1+1+1+1+1+1)/6} = \sqrt{1} = 1$. Model B, $\sqrt{(0+0+0+0+0+12)/6} = \sqrt{2} = 1.41$ — actually different here, but tune the one large miss down to $-2.449$ instead and Model B's RMSE becomes $\sqrt{6/6}=1$, identical to Model A's, while Model A is consistently a little wrong and Model B is almost always exactly right except for one dangerous outlier day. A risk manager cares enormously about which of these two "RMSE = 1" models they're holding.

Worked example 2: grouping residuals to find a pattern

A volatility forecasting model has residuals averaging near zero overall. Split them by realized-volatility regime: on low-vol days ($<15\%$ annualized), mean residual is $+0.2$; on high-vol days ($>35\%$), mean residual is $-4.1$ — the model badly underforecasts exactly when volatility is spiking, which is precisely when a bad forecast is most costly, and this pattern is invisible in the overall mean residual because low-vol days vastly outnumber high-vol days and drag the average back toward zero.

Compare the bell curve above — residuals that are small, symmetric, and centered at zero — to a distribution with a heavy tail on one side; a residual histogram that looks like the second is the visual signature error analysis is looking for.

What this means in practice

Before trusting a live trading model, plot its residuals against time, against volatility regime, against sector, and against prediction confidence, not just its headline RMSE or Sharpe. A model that fails specifically during regime shifts or specifically on illiquid names is a very different (and often much riskier) object than its aggregate score suggests, and that difference only shows up once you actually look at where the errors live.