Why ML Strategies Degrade Live

A research team backtests a model on ten years of history, gets a Sharpe ratio of 2.5, presents it to the investment committee, and gets funded. Eighteen months later the live Sharpe is 0.5, and sometimes negative in the first few months. This gap is common enough that experienced allocators discount every backtest before asking a single question about the strategy — and the discount is earned, because the same handful of mechanisms explain most of the shortfall, over and over, across firms.

The usual suspects

Overfitting to noise. A model with enough parameters, tested against enough feature combinations, will find patterns in historical data that are pure coincidence — a stock's returns correlated with a competitor's CEO's Twitter follower count for no causal reason, discovered because thousands of candidate features were tried until one fit. The more features and model variations a research process tries, the more of the backtest's apparent skill is actually the residue of trying many things and keeping the one that happened to work (see Backtest Overfitting).

Look-ahead bias, subtly. Point-in-time data errors are the classic culprit — using a company's finalized, restated financials instead of what was actually reported on the day, or using a feature that was only available with a lag the backtest didn't apply. These leaks are rarely obvious; a single mislabeled timestamp in a data pipeline can inflate a backtest's Sharpe by a full point without any code that looks wrong on inspection.

Regime dependence. A model trained entirely on a low-rate, low-volatility decade learns relationships specific to that regime — a momentum model trained through 2010-2019 learns patterns tuned to a period without a sustained rate-hiking cycle, and its live period starting in 2022 is a different macro world the training data never saw. The model isn't wrong, exactly; it's answering a question about a market that no longer exists.

Crowding. If a signal is genuinely real and not too hard to discover, other quant shops find it too. A signal traded by one fund earns its full historical edge; the same signal traded by twenty funds gets arbitraged toward zero as they all buy the same cheap stocks and sell the same expensive ones, moving the price before any one fund's model captures the historical spread.

Worked example

A team backtests a 44-feature tree model on 8 years of daily cross-sectional equity data, testing roughly 60 feature-set and hyperparameter variations along the way before settling on the best one by in-sample Sharpe. The chosen configuration shows a backtest Sharpe of 2.4. Using the deflated Sharpe ratio correction, which adjusts for exactly this kind of multiple testing, the probability that a Sharpe this high arose purely from selecting the best of 60 trials on noisy data — rather than from real skill — turns out to be uncomfortably high; a haircut consistent with published deflation studies would knock the expected live Sharpe down to somewhere in the 0.6-1.0 range even before accounting for regime shift or crowding. The team's actual live Sharpe over the following two years came in at 0.7 — squarely inside that deflated range, and a useful illustration that the "surprise" wasn't really a surprise if the multiple-testing correction had been applied honestly before funding.

What this means in practice

Serious research processes apply a haircut to a backtest before it ever reaches an allocator: deflated Sharpe ratios that penalize for the number of trials run, out-of-sample testing on data genuinely held back until the model is finalized (see In-Sample and Out-of-Sample Protocol), and combinatorial validation methods that estimate the probability the best-looking backtest is actually the best strategy versus just the luckiest one (see Combinatorially Symmetric Cross-Validation and PBO). None of this eliminates the live gap — regime shift and crowding aren't statistical artifacts and can't be corrected away in a backtest — but it prevents the more embarrassing version of the gap, the one caused by a research process fooling itself.