VaR Backtesting and the Kupiec Test

Most risk models cannot be checked. If a model says a bond is worth 98.4 and it trades at 98.4, you have learned nothing, because the market told you the answer. Value at Risk is different, and that difference is why it took over risk management. A VaR model makes a statement the future can contradict: tomorrow's loss will exceed this number on only one day in a hundred. Wait a hundred days, count how often it happened, and the model has either kept its word or it has not.

That counting exercise is backtesting, and the awkward part is that the answer is never clean. You promised one breach in a hundred and got three. Broken model, or an unlucky quarter? Kupiec's test turns that question into a number.

An analogy before any symbols

Think of a weather forecaster who says "30% chance of rain" every morning. You cannot grade any single forecast: it either rained or it did not, and both are consistent with 30%. What you can do is collect a hundred such mornings and count. If it rained on roughly thirty, the forecaster is calibrated. If it rained on seventy, the forecaster is broken, and no amount of protesting that each individual forecast was defensible will save them.

A VaR model is exactly this forecaster with a much smaller promised probability. At 99% confidence it says "there is a 1% chance today's loss exceeds $4m." Over 250 trading days you expect about two and a half such days. Backtesting is the discipline of counting them honestly and knowing how much deviation from two and a half is normal.

The catch is that rare events are hard to count. A forecaster promising 30% gives you plenty of rain to grade. A model promising 1% gives you two or three data points a year, and two or three data points do not settle arguments.

Exceptions, and why they are a coin flip

Fix a confidence level. Write $p$ for the promised breach probability, so $p = 0.01$ at 99% confidence. Each trading day, compare the actual profit and loss with the VaR number you published the evening before. Define the exception indicator

$I_t = \begin{cases} 1 & \text{if the day's loss exceeded VaR} \\ 0 & \text{otherwise.}\end{cases}$

In words: mark each day with a 1 if the model was wrong and a 0 if it was right. Nothing else matters, not the size of the loss, not the direction of the market. A loss one dollar past the line and a loss ten times the line both count as a single 1. That is a real limitation and we will return to it.

If the model is correctly calibrated, each $I_t$ is a coin flip that lands on 1 with probability $p$, and the flips are independent. The total number of exceptions over $N$ days,

$x = \sum_{t=1}^{N} I_t,$

is then a binomial count with $N$ trials and success probability $p$. Its expected value is $Np$ and its standard deviation is $\sqrt{Np(1-p)}$. At $N = 250$ and $p = 0.01$ you expect $2.5$ exceptions with a standard deviation of about $1.57$, which already tells you that anywhere from 0 to 6 is unremarkable.

Because $p$ is small and $N$ is large, that binomial is almost exactly a Poisson count with mean $\lambda = Np$. Drag the rate below to $2.5$ and you are looking at the exact distribution of a year's exception count under a perfectly calibrated 99% model. Notice how flat the top is, and how much probability sits at 0, 1 and 2:

Kupiec's proportion-of-failures test

Kupiec (1995) turns the comparison into a likelihood ratio. You have two competing stories for the data. The null says the true breach rate is exactly the promised $p$. The alternative says it is whatever the data suggests, namely the observed rate $\hat\pi = x/N$. Compute how likely the observed count is under each story and compare:

$\text{LR}_{\text{POF}} = -2\ln\!\left[\frac{(1-p)^{N-x}\,p^{x}}{(1-\hat\pi)^{N-x}\,\hat\pi^{x}}\right].$

In words: the top of the fraction is the probability of the data if the model is telling the truth; the bottom is the probability of the data under the best possible breach rate. If those two are close, the ratio is near 1, the logarithm is near 0, and the model survives. If the honest rate fits far better, the statistic gets large. POF stands for proportion of failures.

Under the null this statistic follows a chi-squared distribution with one degree of freedom. The critical values worth memorising are 3.84 at the 5% significance level and 6.63 at 1%. Exceed them and you reject the model.

Worked example: a model that fails

A desk runs a 99% one-day VaR model. Over $N = 250$ days it recorded $x = 8$ exceptions. Expected: 2.5. Is 8 too many?

Step 1, the observed rate. $\hat\pi = 8/250 = 0.032$, so the model breached on 3.2% of days rather than the promised 1%.

Step 2, the log-likelihood under the null ($p = 0.01$, with $N - x = 242$ quiet days):

$242 \ln(0.99) + 8\ln(0.01) = -2.4322 - 36.8414 = -39.2735.$

Step 3, the log-likelihood under the alternative ($\hat\pi = 0.032$):

$242 \ln(0.968) + 8\ln(0.032) = -7.8706 - 27.5362 = -35.4068.$

Step 4, the statistic. $\text{LR}_{\text{POF}} = -2(-39.2735 + 35.4068) = 7.73$.

That sits above both 3.84 and 6.63, so the model is rejected at the 1% level. The p-value is about 0.005: if the model really did breach on 1% of days, a count this extreme would show up in roughly one backtest in two hundred. This is not bad luck. Something in the model, most likely a volatility estimate that reacts too slowly, is understating risk.

Worked example: a model that looks wrong but is not

The same desk, a different year. This time $N = 250$ and $x = 1$. Only one exception against an expectation of 2.5, so the model appears far too conservative, tying up capital it does not need.

Step 1. $\hat\pi = 1/250 = 0.004$.

Step 2, null log-likelihood. $249\ln(0.99) + \ln(0.01) = -2.5025 - 4.6052 = -7.1077$.

Step 3, alternative log-likelihood. $249\ln(0.996) + \ln(0.004) = -0.9980 - 5.5215 = -6.5195$.

Step 4. $\text{LR}_{\text{POF}} = -2(-7.1077 + 6.5195) = 1.18$, comfortably below 3.84, with a p-value of about 0.28.

You cannot reject the model. The honest conclusion is not "the model is fine" but "one year of data cannot tell the difference between a 1% model and a 0.4% model." This is the single most important practical fact about VaR backtesting: the test has very little power. A model whose true breach rate is 0.5%, meaning it overstates risk by a factor of two, will pass a 250-day Kupiec test the overwhelming majority of the time. Rejections happen when a model is badly wrong, not when it is merely wrong.

The Basel traffic light

Regulators turned the same binomial arithmetic into a table. Using 250 days at 99% confidence, the Basel Committee asks: if the model were correct, how likely is a count this small or smaller?

Exceptions in 250 days	Cumulative probability	Zone	Capital multiplier add-on
0 to 4	up to 89.22%	green	0.00
5	95.88%	yellow	0.40
6	98.63%	yellow	0.50
7	99.60%	yellow	0.65
8	99.89%	yellow	0.75
9	99.97%	yellow	0.85
10 or more	99.99%+	red	1.00

The base capital multiplier is 3, so a bank in the red zone pays 4, a 33% increase in market-risk capital. Note that the eight exceptions from the first worked example land in yellow, not red, even though Kupiec rejects at 1%. The regulator is deliberately more forgiving than the statistician, because it is grading a bank rather than a model.

What this means in practice

• Use the longest clean history you can. Power scales with $N$, so moving from 250 days to 1000 matters far more than the choice of statistic.
• Backtest at 95% too. A 95% model produces roughly 12 exceptions a year instead of 2.5, giving you five times the data. Many desks diagnose at 95% and report at 99%.
• Split the P&L. Clean backtesting compares VaR against a hypothetical P&L from holding yesterday's positions unchanged; dirty backtesting uses the actual P&L including intraday trading and fees. Only the clean version tests the model; the dirty version tests the model plus the traders.
• Read the dates, not just the count. If the exceptions all fall in one week, or all involve the same risk factor, you have found a specific defect that no aggregate statistic will name for you. Then pair Kupiec with a test of independence, which is what Christoffersen's conditional coverage test adds.

Practice

1. A 95% VaR model is backtested over 500 days and records 38 exceptions. Compute $\hat\pi$, then the Kupiec statistic, and state your conclusion at the 5% level.
2. Using the Basel table, explain why a bank with 5 exceptions faces a penalty while a bank with 4 does not, even though the difference between them is a single day. What does that say about threshold-based regulation?
3. Two models are backtested over 250 days at 99%. Model A has 3 exceptions on scattered dates. Model B has 3 exceptions on three consecutive days in March. Kupiec gives both the same score. Which would you rather run, and what statistic would separate them?
4. You are asked to prove your VaR model is not too conservative. Roughly how many days of backtest history would you need to reliably detect a true breach rate of 0.5% against a promised 1%? Argue from the standard deviation $\sqrt{Np(1-p)}$ rather than computing exactly.