Backtesting ML Risk Models

Backtesting a classical risk model like GARCH or a simple VaR formula is relatively forgiving — there are only a handful of parameters, so there's limited room for the model to have secretly memorized the test period. An ML risk model with thousands of parameters and dozens of engineered features has far more places to hide overfitting, and several of them are invisible unless you specifically go looking. Backtesting an ML risk model properly means checking for a longer list of failure modes than a classical model requires, not just running the same walk-forward test and calling it done.

An analogy: grading a student who had access to a similar exam

A student who studied the actual exam questions in advance will pass with flying colors — and tell you nothing about whether they understand the material. An ML risk model backtest has exactly this risk multiplied several times over: through features that accidentally encode information from the future (leakage), through a model with enough capacity to essentially memorize specific historical events rather than learn general patterns (overfitting), and through hyperparameters that were quietly tuned by peeking at the same test data being used to "validate" the model (test-set contamination). Each of these makes the backtest look better than the live model will actually perform, for a different underlying reason.

The specific things to check

Feature-level leakage: does every input feature use only information genuinely available at the time the prediction is made, with no forward-looking window, restated data, or vendor timestamp that lags the true event time? Overfitting via capacity: does performance degrade sharply between training-period fit and a genuinely held-out test period, especially the most recent held-out slice rather than a slice interleaved with training data (interleaving can leak information through nearby time points)? Hyperparameter contamination: were the model's hyperparameters tuned using the same data being used to evaluate final performance, even indirectly through repeated trial-and-error? Regime coverage: does the backtest period include the kinds of stress regimes (rate shocks, liquidity crunches, correlation spikes) the model will eventually need to handle live, or only a run of relatively calm markets?

Worked example: a leakage bug hiding in a feature

A tail-event classifier for a credit portfolio is backtested with a walk-forward split and shows an impressively low false-negative rate, catching nearly every historical default. On investigation, one feature, "days since last rating downgrade," turns out to use the rating agency's publication date rather than the effective date — and agencies sometimes backdate the effective date when correcting errors, so the feature occasionally encoded information that, at the true point in time, hadn't been published yet. Recomputed with strictly point-in-time-correct data, the backtested false-negative rate roughly doubles — the corrected number is the one that actually reflects live behavior.

What this means in practice

Every ML risk model backtest needs an explicit leakage audit of each feature's timing, a genuinely held-out final test slice that was never touched during hyperparameter tuning, and a check that the backtest period includes stress regimes rather than only calm ones. Skipping any of these doesn't just risk a somewhat-optimistic number — because ML models have so much more capacity to fit spurious patterns than classical ones, the gap between a flawed backtest and true live performance tends to be much larger than for a simple parametric model.