Regression Error Metrics

Two forecasters both predict tomorrow's return. Forecaster A is off by a tiny amount on 99 days and wildly wrong on one day. Forecaster B is off by a small, steady amount every day, never wild but never close either. Which is "more accurate"? There's no single answer — it depends on whether a rare huge miss or a constant small miss hurts you more, a business decision, not a mathematical one. Regression error metrics are the different ways of turning "how wrong were the predictions" into one number, each encoding a different answer to that question.

Every metric below starts from the same raw material: for each of $n$ predictions, the residual $e_i = y_i - \hat{y}_i$, the actual value minus the predicted value. A residual of $0$ is a perfect prediction; the sign tells you whether the model over- or under-shot.

Mean Absolute Error — treats every miss proportionally

$$\text{MAE} = \frac{1}{n}\sum_{i=1}^{n} |e_i|$$

In words: take the absolute size of every miss, ignoring its sign, and average them. A miss of $4 counts exactly twice as much as a miss of $2. MAE is in the same units as $y$ itself, which makes it directly readable: "the model is off by $0.35 on average."

Root Mean Squared Error — punishes big misses disproportionately

$$\text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n} e_i^2}$$

In words: square every miss before averaging (which makes large misses count far more than small ones, since squaring a miss twice as big makes it four times as costly), then take the square root at the end to bring the units back to match $y$. Because of the squaring, RMSE is always $\geq$ MAE, and the gap between them is a direct readout of how uneven the errors are — a close RMSE/MAE ratio means errors are all similar in size, a wide gap means a few outliers dominate.

MAPE — makes errors comparable across different scales

$$\text{MAPE} = \frac{100}{n}\sum_{i=1}^{n} \left| \frac{e_i}{y_i} \right|$$

In words: express each miss as a percentage of the true value, then average those percentages. This lets you compare forecast quality on a $5 stock and a $500 stock on the same scale, but it blows up or becomes meaningless whenever $y_i$ is near zero — a stock return forecast that's off by $0.01 on a day the true return was $0.02 registers as a "50% error" even though both numbers are tiny.

Drag the exponent and watch how the curve near zero stays gentle while the curve far from zero steepens fast — that steepening is exactly what squaring does to RMSE relative to MAE: small errors are barely amplified, large errors are amplified a lot.

Worked example 1: same average miss, different RMSE

Model A's residuals across 5 days: $1, 1, 1, 1, 1$ (all misses are exactly $1). Model B's residuals: $0, 0, 0, 0, 5$ (perfect four days, one $5 disaster).

MAE for both: A gives $\frac{1+1+1+1+1}{5} = 1.0$. B gives $\frac{0+0+0+0+5}{5} = 1.0$. Identical — MAE says these models are equally accurate on average.

RMSE tells a different story. A: $\sqrt{\frac{1+1+1+1+1}{5}} = \sqrt{1} = 1.0$. B: $\sqrt{\frac{0+0+0+0+25}{5}} = \sqrt{5} \approx 2.24$. RMSE more than doubles for B, correctly flagging that a model with one catastrophic miss is riskier than one with steady small misses, even though the average absolute error is the same. If a $5 miss would trigger a margin call and a $1 miss would not, RMSE is measuring the thing that matters; MAE would tell you nothing is wrong.

Worked example 2: MAPE breaking down near zero

A model forecasts three days of returns: true values $y = 2.0\%, 0.1\%, -3.0\%$, predictions $\hat{y} = 1.8\%, 0.4\%, -2.7\%$. Residuals: $0.2, -0.3, -0.3$ points. MAE $\approx 0.267$ points — a sensible, stable number. MAPE: $\left|\frac{0.2}{2.0}\right|=10\%$, $\left|\frac{-0.3}{0.1}\right|=300\%$, $\left|\frac{-0.3}{-3.0}\right|=10\%$, averaging to $\approx107\%$. One day where the true return happened to be tiny dominates the whole metric and makes an otherwise decent model look catastrophic — exactly why MAPE is avoided for return forecasting, where true values routinely sit near zero.

What this means in practice

A backtest reporting only RMSE can hide a model that's usually excellent and occasionally disastrous — exactly the failure mode a risk manager needs to see, so RMSE is standard where tail losses matter. One reporting only MAE can hide that a model is quietly unreliable on the days that matter most, since huge misses get diluted by easy ones. Reporting both, alongside a residual plot, is standard practice precisely because neither number alone tells the full story.