Error Analysis and Slice-Based Evaluation

A volatility forecasting model reports a respectable overall error metric across a year of daily data. Deployed live, it's fine 90% of the time and badly wrong exactly during the days that matter most — the handful of high-volatility regime shifts where a good forecast would have been most valuable. The aggregate metric never showed this, because those days are a small fraction of the total and get averaged away. Error analysis is the practice of deliberately breaking the test set into meaningful slices — by regime, by asset class, by time of day, by any variable that might interact with performance — and checking accuracy slice by slice instead of trusting one overall number.

Why aggregates hide the interesting failures

An overall metric is a weighted average across every case in the test set, and averages are exactly the statistic most able to hide a bad subgroup behind a good majority. If a model handles 95% of cases (low-volatility, calm markets) very well and 5% of cases (regime transitions) very badly, the overall number can still look strong, because that 5% barely moves a weighted average. But those cases may be precisely the ones with outsized financial consequences — the days a hedge actually needs to work, or the trades where a mispriced risk model is most costly. Slicing means computing the same metric separately within each candidate subgroup and comparing them, rather than accepting one number for the whole dataset.

Worked example: slicing a volatility model by regime

A model's overall mean absolute error (MAE) on one year of daily volatility forecasts is 2.1 volatility points. Slicing the test set by realized regime (using a simple threshold on trailing realized volatility) gives: calm days (80% of the sample) — MAE 1.4; elevated days (15% of the sample) — MAE 3.6; crisis days (5% of the sample) — MAE 9.8. The overall average of 2.1 is dominated by the 80% of calm days and completely obscures that the model is roughly 7x worse, in absolute terms, exactly on the crisis days a risk manager cares most about. This single slicing exercise — one line of grouping code — turns "the model is decent" into "the model needs a regime-specific fix or a fallback rule for crisis days," a conclusion the aggregate number alone would never have surfaced.

What this means in practice

Slice a test set along every dimension plausibly linked to difficulty or business importance: asset class, liquidity tier, time of day, market regime, data vintage, or any categorical feature available at prediction time. Look specifically for slices that are both large enough to matter and where performance diverges sharply from the aggregate — those are the ones worth a targeted fix, whether that's a separate model, an added feature, or simply flagging predictions in that slice as lower-confidence. Error analysis is qualitative as much as quantitative: reading through a sample of the worst individual errors in the worst-performing slice often reveals a specific, fixable pattern (a missing feature, a data quality issue, a mislabeled edge case) that no summary statistic would name directly.