Cross-Validation When Samples Are Dependent

Standard k-fold cross-validation randomly shuffles all the rows into folds, trains on some, and tests on the rest — a procedure that works beautifully when every row is an independent draw from the same process. Daily stock returns aren't independent draws: yesterday's return is correlated with today's, a feature built from a 20-day rolling window overlaps heavily with the same feature computed one day later, and many stocks move together on the same macro day. When rows this dependent get shuffled into random folds, a training fold ends up containing information that's almost a copy of what's sitting in the test fold, and the model's cross-validated score looks far better than its true out-of-sample performance will be.

Why random shuffling leaks information

Consider a feature built from a 20-day trailing average of returns, used to predict tomorrow's return. Day $t$'s feature and day $t+1$'s feature share 19 of their 20 underlying days of data. If day $t$ lands in the training fold and day $t+1$ lands in the test fold, the model effectively gets to "see" almost the same information it's being tested on, just relabeled by one day. This is a form of data leakage — not because any single row is duplicated exactly, but because the dependence structure between nearby rows means a random split doesn't actually separate what the model learns from what it's evaluated on.

$$\text{Corr}(x_t, x_{t+1}) \gg 0 \quad \Rightarrow \quad \text{random fold assignment leaks information across the split}$$

In plain English: whenever a feature (or the target) is autocorrelated across time, any cross-validation scheme that ignores time ordering when assigning rows to folds will overstate performance, because it's testing the model on data that's statistically close to data it trained on.

Worked example: purging and embargo

A quant builds a 5-day-ahead return prediction model using 60-day rolling features. Instead of random k-fold, they use a time-ordered split: fold boundaries follow calendar order, and around each boundary they purge any training observation whose 60-day feature window overlaps the test period, and add an embargo of a few days after the test period before resuming training data, so no training row's label window bleeds into the test window either. On a dataset of 1,000 trading days split into 5 folds, this might drop roughly 60 days of training data around each of the 4 internal boundaries (about 240 days total) that would otherwise leak — a real reduction in usable training data, but one that makes the resulting cross-validated Sharpe ratio a genuine estimate rather than an inflated one. In practice, a signal that shows a cross-validated Sharpe of 1.8 under naive random k-fold often drops to something like 0.9–1.2 under a properly purged, embargoed, time-ordered split — the gap is the leakage the naive scheme was hiding.

The explorer above generates autocorrelated paths — notice how neighboring points in time look similar to each other by construction; that's the exact structure a random fold split fails to respect.

What this means in practice

Whenever observations have time dependence, cross-sectional overlap (like many stocks sharing a macro shock on the same day), or overlapping feature windows, use a time-ordered split with purging and an embargo rather than random k-fold, and treat any cross-validated performance number computed with plain random shuffling on such data as optimistically biased until proven otherwise.