Designing the Lookback Window

A "20-day moving average" feature sounds like a fixed, objective quantity, but the 20 is a choice, and a different choice produces a meaningfully different feature. A 5-day lookback reacts fast to new information but is noisy — a single unusual day swings it a lot. A 100-day lookback is smooth and stable but sluggish — it's still half-reflecting a regime that ended two months ago. Lookback window design is the deliberate process of choosing that window length to match the actual dynamics of what's being predicted, rather than defaulting to whatever number a textbook or a previous project happened to use.

The trade-off in one picture

A rolling feature computed over a window of length $w$ averages out noise proportional to $1/\sqrt{w}$ — doubling the window roughly shrinks noise by a factor of $1/\sqrt{2}$, a diminishing return. But a longer window also blends together information from further in the past, which is only useful if the underlying process is genuinely persistent over that horizon; if the true signal decays faster than the window length, a long window mixes in stale, irrelevant history that dilutes the current signal rather than clarifying it.

$$w^{*} \approx \text{(persistence of the underlying signal)}$$

In plain English: the right window length should roughly match how long the thing you're measuring actually stays relevant — a fast-decaying signal (like short-term order-flow imbalance) wants a short window; a slow-moving one (like a company's long-run profitability trend) wants a long one, and using the wrong length either drowns the signal in noise or drowns it in stale history.

Worked example: choosing a volatility lookback by trial

A team predicting next-day volatility tests trailing realized-volatility features computed over 5, 20, and 60 days, each fed into an otherwise identical model, and measures out-of-sample correlation with next-day realized volatility. The 5-day feature achieves a correlation of 0.31 (noisy, reacts to single-day spikes that don't persist), the 20-day feature achieves 0.44 (the best balance), and the 60-day feature drops to 0.38 (too smooth, lagging behind genuine regime shifts by weeks). This single sweep — same target, same model, only the window length varied — is usually enough to locate roughly where the trade-off peaks, and it's worth re-running periodically, since the right window length can itself shift as market dynamics change (a market that's grown choppier may reward a shorter window than it did five years earlier).

Watching the explorer above with different mean-reversion speeds is a good intuition pump: a fast-reverting process (analogous to a short-persistence signal) needs a short lookback to track it; a slow-reverting one tolerates, and benefits from, a longer one.

What this means in practice

Treat the lookback window as a hyperparameter to be tuned on validation data (via a walk-forward sweep, not a single in-sample fit), not a convention to be inherited unquestioned. It's also worth testing several window lengths as separate features fed to the same model rather than picking only one — a short and a long moving average together often outperform either alone, because they capture both the fast and slow components of the same underlying process.