Label Overlap and Effective Sample Size

Build a training set by computing a 20-day forward return as the label for every single trading day over five years, and you get roughly 1,250 rows. It looks like 1,250 independent observations. It isn't. The label for Monday (the return from Monday to four weeks later) and the label for Tuesday (the return from Tuesday to four weeks and a day later) share 19 of their 20 underlying days almost entirely — they are telling you nearly the same thing about nearly the same period. Treating this dataset as if it contained 1,250 independent pieces of evidence dramatically overstates how much the model has actually learned, and understates how uncertain any accuracy estimate really is.

Why overlap inflates apparent sample size

When forward-looking labels are computed with a rolling window shorter than the gap between rows (here, a 20-day window computed every 1 day), each label overlaps with the previous label's window by $19/20 = 95\%$. The effective sample size — the number of genuinely independent pieces of information in the data — is much smaller than the raw row count. A common rule of thumb approximates the effective sample size as

$$n_{\text{eff}} \approx \frac{n}{h} ,$$

where $n$ is the raw number of rows and $h$ is the label horizon in the same units as the row spacing (here, days). In plain English: if your label looks 20 days ahead and you compute a new label every single day, you effectively have about one-twentieth as many genuinely independent observations as raw rows, because each label is mostly reporting the same underlying 20-day return as its close neighbors.

Worked example: five years, one horizon

Suppose you have 1,250 trading days of history and label every day with its 20-day forward return, producing 1,250 rows. Using the rule of thumb, $n_{\text{eff}} \approx 1{,}250 / 20 = 62.5$ — roughly 62 genuinely independent 20-day return observations, not 1,250. A statistical test (or a standard error on the model's estimated accuracy) computed as if $n = 1{,}250$ would be dramatically overconfident: standard errors shrink with $\sqrt{n}$, so treating the sample as 20 times larger than it really is makes an estimate's reported precision roughly $\sqrt{20} \approx 4.5$ times tighter than it should be — enough to make a genuinely marginal, noisy result look like a solid, statistically significant finding.

Consequences beyond statistical tests

Overlap doesn't just distort significance tests — it also breaks the assumption behind standard cross-validation. If a random 80/20 train/test split happens to place Monday's row in training and Tuesday's row (which shares 95% of the same underlying return window) in test, the model isn't really being tested on unseen data; it's effectively being tested on data it already saw almost all of. This is why purged and embargoed cross-validation schemes exist specifically for financial ML — they remove training rows whose label windows overlap with a test period's label window, rather than splitting purely by row index or date.

The tool above shows how sample means from independent draws behave — overlapping labels violate the independence this relies on, which is exactly why a naive standard error computed on overlapping data understates true uncertainty.

What this means in practice

Any time a label is built from a forward-looking window and computed more frequently than that window's length, the effective sample size — not the raw row count — should govern how much confidence to place in a backtest statistic, and cross-validation splits must be purged of overlap with the test period rather than drawn at random.