Your Volatility Model Comparison Is Probably Wrong: Patton on Imperfect Proxies

Suppose you have two volatility forecasting models and you want to know which is better. You run both, you compare each forecast to what actually happened, you compute an average error, and the model with the smaller error wins.

Simple. Obvious. And, as Andrew Patton showed in 2011, potentially completely wrong.

Not wrong because of bad data. Not wrong because of overfitting. Wrong because of a subtle interaction between two facts that everyone in the field already knew separately and nobody had put together:

1. You can never observe true volatility, so you compare forecasts against a proxy (a squared return, the daily high-low range, realized variance from intraday data).
2. You have to choose a loss function, a rule for turning "forecast said 15%, proxy said 22%" into a penalty score.

Patton proved that for many perfectly sensible-looking loss functions, the proxy's noise systematically distorts the ranking, and the model that scores best is not the model that is best. You could reject a genuinely superior forecast in favour of an inferior one, run the numbers as carefully as you like, and never suspect a thing.

The problem: unbiased is not the same as harmless

The comforting thought that had lulled the field to sleep goes like this. Yes, the squared return is a noisy measure of true volatility. But it is an unbiased one: on average, over many days, it equals the truth. And noise that averages out to zero surely just adds a bit of imprecision to the comparison without changing which model wins. Right?

Wrong, and the reason is a piece of mathematics that trips people up everywhere. Noise averages out only when it passes through a linear operation. The moment you feed a noisy input through a nonlinear function, the noise does not cancel. It leaves a residue, and the residue is not the same for every forecast.

Most loss functions are nonlinear. Anything involving ratios, logarithms, percentages, absolute values. These are exactly the loss functions people gravitate toward for volatility, because volatility is a positive quantity that varies over orders of magnitude, so measuring errors in percentage terms feels natural.

And that natural instinct is precisely the trap. Patton showed that with a loss function like absolute error, or percentage error, or the log of the ratio, the ranking of two models can genuinely flip depending on how noisy your proxy is. The distortion is not random. It is systematic. It reliably favours certain kinds of forecast over others for reasons that have nothing to do with forecasting skill.

The key idea via analogy: the scale with a wobbly needle

Imagine judging a weight-guessing contest with a bathroom scale whose needle jitters randomly by a few kilograms. The scale is unbiased: over many weighings it is right on average.

Contestant A always guesses close to the true weight. Contestant B always guesses low. If you score by simple squared error, the jitter hurts both contestants equally, and A wins, correctly.

But now score them by percentage error instead. The jitter is proportionately much more damaging when the true weight is small. Contestant B, who systematically guesses low, gets an artificial advantage in exactly the situations where the noise blows up the percentage denominator. Score the contest for long enough and B can win.

The scale was fine. The scoring rule was not. That is Patton's paper.

The answer: only certain loss functions are safe

The valuable part of the paper is that Patton does not merely raise the alarm, he solves the problem. He derives the exact mathematical conditions a loss function must satisfy for the ranking of forecasts to be robust: that is, for the ranking you get using a noisy proxy to be the same ranking you would get if you could see true volatility.

The condition is restrictive, and the surviving family is small. Two members of it are already familiar:

• Squared error on the variance (penalise the squared difference between forecast variance and the proxy). Safe.
• The QLIKE loss, a particular ratio-and-logarithm combination that comes out of the Gaussian likelihood. Safe.

And an uncomfortable number of popular choices are not safe, including mean absolute error, mean absolute percentage error, and various R-squared-style measures that people had been using routinely for years. A meaningful chunk of the published volatility forecasting literature had been graded on a bent scale.

Patton also shows that even with a robust loss function, a noisier proxy still costs you: it reduces your power to detect a real difference between models. So better proxies still matter. Realized variance beats the squared return, even when the loss function is safe. Get both right: a robust loss function and the best proxy you can build.

Why it mattered

• It rewrote the rules of forecast comparison. Papers written after 2011 that compare volatility models are expected to use a robust loss function, and reviewers ask about it. QLIKE and squared error are now the standard pair.
• It cast doubt on earlier results. Any historical comparison using absolute or percentage errors is potentially compromised. That is a lot of papers.
• It generalises well beyond volatility. The lesson, that when you evaluate predictions of an unobservable quantity your loss function must be chosen to be robust to the noise in your proxy, applies anywhere you are forecasting a latent variable, which in finance is most of the interesting cases.
• It is a rare instance of a paper being both a warning and a complete cure. It tells you the problem and hands you the exact set of loss functions that solve it.

The honest limitations

• The robust family is narrow. Sometimes you genuinely care about percentage errors, because that is how your economic problem is denominated. Patton's result tells you that you cannot use that loss for ranking models without risk, which can be genuinely inconvenient.
• Conditional unbiasedness is assumed. The theory requires the proxy to be conditionally unbiased. Realized variance contaminated by microstructure noise is not, strictly speaking, and the guarantees weaken accordingly.
• Robustness is about ranking, not about magnitude. A robust loss function gets the ordering right. It does not tell you how much better the winning model is in any economically meaningful sense.
• It says nothing about which model to use. The paper is about how to compare. Whether the winner of a properly conducted comparison is a good model for your actual purpose, hedging, risk capital, options pricing, is a separate question with a separate answer.

The one-line takeaway

Because true volatility is invisible, you grade forecasts against a noisy stand-in, and Patton proved that with most of the scoring rules people naturally reach for, the noise systematically corrupts the ranking and can crown the worse model, leaving only a small family of safe loss functions, of which squared error and QLIKE are the two you should actually use.