Data Leakage in Machine Learning
When information the model would not truly have at prediction time sneaks into training, the backtest looks brilliant and the live strategy fails — the single most common reason ML trading models don't survive contact with markets.
Prerequisites: Look-Ahead Bias, Cross-Validation for Time Series
Data leakage is when your model, during training, gets to peek at information it would not actually have at the moment it makes a prediction. The result is a model that scores wonderfully in the backtest and collapses live, because the crutch it leaned on, tomorrow's data, the future mean, the answer smuggled into a feature, simply isn't there in real time. Leakage is the number-one reason machine-learning trading models fail, and it is insidious precisely because the leaked model looks like your best one.
The clean definition: a feature or a validation procedure leaks if it uses information not available at prediction time. Everything you feed the model must be knowable strictly before the moment being predicted. When that rule is broken, the backtest is measuring a fantasy.
Leakage = training on information you would not have in production. The tell is a backtest that looks too good, then a live result that reverts to noise. The cure is a discipline: at every step ask "would I actually know this at prediction time?"
The common channels
| Type of leakage | What happens | The fix |
|---|---|---|
| Target leakage | A feature is a proxy for, or built from, the label itself | Drop any feature computed from the outcome window |
| Preprocessing leakage | Scaler, PCA, or feature selection fit on the full sample | Fit every transform inside the training fold only |
| Temporal / look-ahead | A feature uses data timestamped after the prediction | Lag features; align to information actually released |
| Train-test contamination | Overlapping labels or duplicates split across folds | Purged & Embargoed Cross-Validation on labeled series |
| Survivorship | Universe built from names that exist today | Use a point-in-time universe, see Survivorship Bias |
Two of these deserve special attention in finance. Temporal leakage is the classic Look-Ahead Bias: using a quarterly earnings number on the quarter-end date, when it was not actually reported for another six weeks. And train-test contamination through overlapping labels is subtle enough that it gets its own machinery, Purged & Embargoed Cross-Validation, because ordinary Cross-Validation for Time Series freely places informationally-linked observations on opposite sides of a split.
Worked example: the scaler that saw the future
You standardize a feature before training, subtracting its mean and dividing by its standard deviation, and you fit that scaler on the entire dataset before splitting into train and test. This looks harmless. It is leakage.
Say the feature is monthly and the full-sample mean is 10. Your test period is a bull run where the feature averaged 14, and your training period averaged 8. When you z-score using the full-sample mean of 10, every training observation is shifted using knowledge that the future (the test period) ran hot. Concretely, a training value of 8 becomes , a negative z-score, precisely because the model was told, through the mean, that later values would be higher. The model quietly learns to expect the regime it is about to be tested on.
Do it correctly: fit the scaler on the training data alone. Now the mean is 8, the training value of 8 maps to a z-score of 0, and the test values, which the scaler has never seen, get transformed with only past information. The backtest score drops, often by several points of accuracy, and that lower number is the honest one. The gap between the two is pure leakage.
Any transform that learns from data, scaling, PCA, feature selection, imputation, target encoding, must be fit on the training fold and only applied to the test fold. Fitting it on the full sample before splitting is the most common leak of all, and it hides in a single innocent-looking line of preprocessing code.
How to catch it
Leakage rarely announces itself. Some reliable smells and checks:
- A suspiciously high score. If a market model cross-validates far above what the efficiency of the asset should allow, assume leakage until proven otherwise.
- A feature that is too predictive. Rank feature importance; if one feature dominates implausibly, check whether it was built from the label or from future data.
- The gap test. Retrain with a strict time gap between features and label. If performance craters, temporal leakage was inflating it.
- Fold-order sensitivity. If shuffling versus time-ordering the folds changes the score a lot, information is crossing the split it should not.
Build the whole preprocessing chain as a single pipeline object that is fit only on training data and then transforms the test data. Frameworks make this a one-liner, and it structurally prevents the most common preprocessing leaks, you cannot accidentally fit a scaler on the test set if the pipeline forbids it.
The bottom line
Leakage turns a backtest into a mirror that flatters you. The remedies are not exotic, they are discipline: lag every feature to when its information was truly available, fit every transform inside the fold, purge and embargo overlapping labels, build a point-in-time universe, and grade the final model with Nested Cross-Validation so tuning never touches the reported score. A model with a modest, honest, leak-free backtest is worth infinitely more than a spectacular one built on borrowed future, because only the honest one has a chance of surviving live. Every inflated number here feeds directly into the broader Backtest Overfitting problem.
Related concepts
Practice in interviews
Further reading
- Kaufman et al., Leakage in Data Mining: Formulation, Detection, and Avoidance
- López de Prado, Advances in Financial Machine Learning (Ch. 7)
- Kapoor & Narayanan, Leakage and the Reproducibility Crisis in ML-based Science