Backtesting Expected Shortfall

Testing whether a VaR model is any good is refreshingly simple: count how often actual losses exceeded the reported VaR, and compare that exception count to what the confidence level promised. A 99% VaR should be breached about 1% of the time; too many breaches and the model understates risk, too few and it is overly conservative. Expected shortfall (ES) has no equivalent trick, because it is not a threshold you either breach or do not, it is an average of everything beyond a threshold, and you cannot directly observe "the average of a set of events that mostly didn't happen."

The analogy before any symbols

Checking whether a smoke alarm goes off often enough relative to actual fires is a simple counting exercise. Checking whether the alarm's volume is calibrated correctly, loud enough to reflect how severe the average fire actually turns out to be, is a completely different and harder question, because you only get to measure volume during the rare events when the alarm actually sounds, and you need to know not just that a fire happened but how bad it was on average across many fires. VaR backtesting is the smoke-alarm counting exercise; ES backtesting has to grapple with the harder, second question.

The mechanics

VaR backtesting counts exceptions: on day $t$, define $I_t = 1$ if the realized loss $L_t$ exceeded $\text{VaR}_\alpha$, else $0$. Over $N$ days, the exception rate $\sum I_t / N$ should be close to $1-\alpha$; a Kupiec or Christoffersen test checks whether the observed rate is statistically consistent with that target. This works because a single number, a 0-or-1 indicator, fully summarizes "did today breach the threshold."

ES backtesting cannot use a 0-or-1 indicator, because ES asks about the size of losses beyond the threshold, not just whether the threshold was crossed. The most widely used approach, Acerbi and Szekely's test, compares realized losses on exception days against the model's ES forecast:

$Z = \frac{\sum_{t=1}^N I_t \, \dfrac{L_t}{\text{ES}_{\alpha,t}}}{\sum_{t=1}^N I_t} - 1.$

In words: on every day the VaR threshold was breached, divide the actual loss by that day's forecasted ES, average that ratio across all breach days, and subtract one. If the model is well calibrated, breach-day losses should average out to roughly the forecasted ES, so this ratio should average to about 1, making $Z$ close to zero. A large positive $Z$ means realized tail losses are systematically bigger than the model predicted, the model's ES is too low; a large negative $Z$ means the model is overstating tail severity.

VaR backtesting only checks whether the shaded boundary in the tail sits in the right place; ES backtesting has to additionally ask whether the average depth of the tail beyond that boundary, not just its existence, matches what the model claimed.

Worked example: computing Z on ten breach days

Over 250 days at 97.5% confidence, VaR is breached on 10 days (close to the expected 2.5%). The model's forecasted ES on those days averaged $1,200,000. The actual realized losses on those 10 breach days were, in millions of dollars: 1.1, 1.3, 0.9, 1.5, 1.0, 1.4, 1.1, 1.6, 0.95, 1.25, averaging $\big(1.1+1.3+0.9+1.5+1.0+1.4+1.1+1.6+0.95+1.25\big)/10 = 1.215$, i.e. $1.215m. $Z = 1.215/1.2 - 1 = 0.0125$, close to zero, meaning realized tail losses matched the ES forecast well. This model passes: not only did breaches occur about as often as expected, but their average severity matched the model's own claim about severity.

Worked example: a model that fails on severity, not frequency

Another model also has exactly 10 breaches out of 250 days at 97.5%, so a pure VaR exception-count test would pass it without complaint. But its forecasted ES on breach days was $900,000, while the actual average breach-day loss was $1,550,000. $Z = 1{,}550/900 - 1 = 0.72$, a large positive value: this model gets the frequency of tail events right by luck but severely understates how bad they are on average when they occur. A VaR-only backtest, which only ever asks "was there a breach," would never catch this; only an ES-style severity check reveals it.

What this means in practice

Since Basel's 2019 market-risk framework (FRTB) moved capital requirements from VaR to ES, at 97.5% confidence specifically because ES is a coherent risk measure and VaR is not, ES backtesting of this kind became a regulatory requirement, not just good practice. Desks typically run both a frequency check (are breaches happening about as often as expected) and a severity check like the one above (are breach-day losses the right size on average), because a model can pass either test alone while failing the other.