The Diebold-Mariano Test

Two volatility models have been forecasting the same series for a year. Model A's average error (say, RMSE) came in at 1.20; Model B's came in at 1.35. Model A looks better — but is a gap of 0.15 real evidence that A is the superior forecaster, or could a coin-flip-different sample of the same underlying process have just as easily produced B looking better? Eyeballing two average error numbers and picking the smaller one ignores that both numbers are themselves estimates, computed from a finite, noisy sample of forecast errors, and therefore have their own sampling uncertainty. The Diebold-Mariano test turns "A's error is smaller than B's" into a proper hypothesis test with a p-value, so you know whether the gap is a real, statistically supported difference in forecasting skill.

An analogy: comparing two archers over many rounds

Two archers each shoot at a target, and after 50 arrows, archer A's average distance from the bullseye is slightly better than archer B's. Before declaring A the better archer, you'd want to know: was A consistently a little better on almost every arrow, or did A get lucky on just a couple of arrows while being roughly tied the rest of the time? If the arrow-by-arrow differences are noisy and only occasionally favor A, the average gap might not survive a rematch. The Diebold-Mariano test is this same check applied to forecast errors: look at the period-by-period difference in accuracy, see how consistent and how large that difference really is relative to its own noise, and only then conclude one forecaster is genuinely better.

The test, one symbol at a time

Define a loss differential at each time $t$ as the difference in some error measure (commonly squared error) between the two models' forecasts:

$$d_t = L(e_{A,t}) - L(e_{B,t}),$$

where $L(e_{A,t})$ is model A's loss (e.g., squared error) at time $t$ and $L(e_{B,t})$ is model B's. If A is systematically more accurate, $d_t$ should average out negative (A's losses are smaller); if the models are equally good, $d_t$ should average out near zero. The test statistic is

$$DM = \frac{\bar{d}}{\sqrt{\widehat{\text{Var}}(\bar{d})}},$$

where $\bar{d}$ is the sample average of $d_t$ across all $n$ forecast periods, and $\widehat{\text{Var}}(\bar{d})$ is an estimate of its sampling variance that accounts for the fact that forecast errors are typically autocorrelated over time (today's error and tomorrow's aren't independent), which an ordinary variance formula would understate. In plain English: DM measures how many standard errors the average loss gap is away from zero, properly accounting for the noisiness and time-dependence of that gap — under the null hypothesis of equal forecast accuracy, DM should behave like a standard normal variable, so values beyond roughly $\pm 1.96$ signal a statistically significant difference at the 5% level.

Worked example 1: computing DM by hand

Suppose over 4 days the squared errors are: Model A = $\{0.4, 0.6, 0.3, 0.5\}$, Model B = $\{0.7, 0.5, 0.6, 0.8\}$. The loss differentials are $d = \{-0.3, 0.1, -0.3, -0.3\}$. The sample mean is $\bar{d} = \frac{-0.3+0.1-0.3-0.3}{4} = \frac{-0.8}{4} = -0.2$. Suppose the estimated standard error of $\bar{d}$ (accounting for autocorrelation) is 0.12. Then

$$DM = \frac{-0.2}{0.12} \approx -1.67.$$

Since $|-1.67| < 1.96$, this falls short of standard 5% significance — despite Model A's average loss being noticeably lower in this small sample, there isn't yet enough statistical evidence, given the noise, to declare A significantly better; a larger sample or a more consistent gap would be needed.

Worked example 2: a clearer win

Now suppose a longer sample (250 days) yields $\bar{d} = -0.15$ with a smaller estimated standard error of 0.04 (more data shrinks the standard error). Then

$$DM = \frac{-0.15}{0.04} = -3.75.$$

With $|-3.75|$ well beyond 1.96 (in fact beyond the 0.1% threshold of about 3.29), this is strong evidence that model A is genuinely more accurate, not merely luckier over this sample — a result a desk could act on, for instance by retiring model B or by weighting a forecast combination toward A.

What this means in practice

The Diebold-Mariano test is the standard way to justify replacing one production forecasting model with another, or to defend a forecast-combination weighting scheme (see Forecast Combination and Averaging), rather than relying on "the new model's backtest error looked smaller." It's routinely used to compare volatility forecasts, macro nowcasts, and any two competing time-series models built on the same target and evaluation window.