Cross-validation with overlapping labels leaks the future

Question

You build a signal to predict each day's forward 20-day return. You evaluate it with ordinary random $10$ -fold cross-validation: shuffle all the daily observations, split into $10$ folds, train on nine, validate on one. The cross-validated accuracy looks excellent. Live, the signal disappoints badly.

Explain why this cross-validation is over-optimistic and what resampling scheme fixes it.

Accepted Answer

The core mistake: overlapping labels plus random folds create leakage. Each observation's label is the return over the next 20 days, so consecutive days share almost all of their label window: day t's label and day t+1's label overlap in 19 of 20 days. Random k-fold scatters these overlapping observations across training and validation folds. As a result, a training observation and a validation observation can cover nearly the same future period, so the model is effectively "seeing" the answer to a validation case during training. Add ordinary shuffling and you also train on the future to pre…

Cross-validation with overlapping labels leaks the future

Your answer

Discussion

Learn the concepts

Related questions