Forecast Evaluation and Accuracy Metrics

You've built a model that forecasts tomorrow's volatility, or next quarter's revenue, or next week's spread. It made predictions, time passed, and now you have the actual outcomes. How good was it? "Pretty close most of the time" isn't a number you can compare across models, track over months, or use to decide whether to keep trusting the forecast. Forecast evaluation metrics turn a pile of (prediction, actual) pairs into a single number that summarizes accuracy — but different metrics answer subtly different questions, and picking the wrong one can make a genuinely bad forecaster look good, or vice versa.

An analogy: grading a dart player two different ways

Imagine grading a dart player by two different scoring rules. Rule A: average how far each throw lands from the bullseye. Rule B: average the squared distance from the bullseye, then take the square root at the end. Both rules reward accuracy, but Rule B punishes one wild miss far more than several small misses, because squaring a big number makes it much bigger, while Rule A treats a 10cm miss as simply ten times worse than a 1cm miss, no more. A player who's usually excellent but occasionally throws wildly off will score worse under Rule B than Rule A. Forecast accuracy metrics are exactly these two grading rules applied to prediction errors instead of dart throws — and the choice between them changes which "player" (model) wins.

The metrics, one symbol at a time

Let $e_t = y_t - \hat{y}_t$ be the forecast error at time $t$ — the actual value minus the predicted value. The two workhorse metrics are:

$$\text{MAE} = \frac{1}{n}\sum_{t=1}^{n} |e_t|, \qquad \text{RMSE} = \sqrt{\frac{1}{n}\sum_{t=1}^{n} e_t^2}.$$

MAE (mean absolute error) is the plain average size of the miss, treating every error in proportion to its own size — the "Rule A" grading. RMSE (root mean squared error) squares each error before averaging, then square-roots the result to get back to the original units — this is "Rule B," and because squaring amplifies large errors disproportionately, RMSE is always at least as large as MAE and penalizes occasional big misses much more heavily than many small ones. A third common metric, MAPE (mean absolute percentage error), expresses the average error as a percentage of the actual value:

$$\text{MAPE} = \frac{100\%}{n}\sum_{t=1}^{n} \left| \frac{e_t}{y_t} \right|,$$

which is scale-free (usable to compare accuracy across series measured in different units, like comparing a stock-price forecast to a volume forecast) but breaks down or explodes when actual values are near zero.

Worked example 1: MAE versus RMSE by hand

Suppose a 5-day volatility forecast produced errors (in vol points) of $e = \{0.5, -0.3, 0.2, -0.4, 3.0\}$ — four small misses and one large one on day 5. MAE:

$$\text{MAE} = \frac{0.5+0.3+0.2+0.4+3.0}{5} = \frac{4.4}{5} = 0.88.$$

RMSE:

$$\text{RMSE} = \sqrt{\frac{0.25+0.09+0.04+0.16+9.0}{5}} = \sqrt{\frac{9.54}{5}} = \sqrt{1.908} \approx 1.38.$$

RMSE (1.38) is over 50% larger than MAE (0.88) precisely because of that one large day-5 miss — squaring turned a single error that's about 7x bigger than the others into a term (9.0) that's about 40x bigger, dominating the sum. If you cared equally about all misses, MAE is the honest summary; if a single large miss is disproportionately costly (say, it triggered a margin call), RMSE reflects that better.

Worked example 2: MAPE and its trap

A revenue forecast predicts $10m and $1m for two business lines; actuals come in at $11m and $1.5m. Errors are $1m and $0.5m — MAE treats these as roughly comparable in magnitude. MAPE instead computes $\frac{1}{10} = 10\%$ for the first line and $\frac{0.5}{1} = 50\%$ for the second, correctly flagging that the small business line's forecast was proportionally far worse even though its dollar error was smaller. This is MAPE's strength — but if the second business line's actual had been $0.01m instead, the same $0.5m-scale miss would produce a MAPE reading in the thousands of percent, a distortion purely from dividing by a near-zero denominator, not from any real change in forecast quality.

What this means in practice

The choice of metric should match what actually costs money when the forecast is wrong. A risk desk that suffers most from rare large misses (blown-through VaR, a margin call) should weight RMSE or an even more skew-sensitive metric; a desk that cares about consistent, proportional accuracy across differently-scaled series should lean on MAPE-family metrics. These raw accuracy numbers are also the necessary first step before formally comparing two forecasting models' skill, as in the Diebold-Mariano test (see The Diebold-Mariano Test).