Leakage Tests in Continuous Integration

Data leakage — a model accidentally seeing information from the future during training — is one of the hardest bugs to catch by eye, because a leaky backtest often looks better than a clean one, not obviously broken. Leakage tests in CI are automated checks, run on every code change, that assert specific leakage-prone patterns can't slip through: that a feature's timestamp never postdates its label, that train and test date ranges never overlap, and that any rolling statistic uses only data available up to that point in time.

These tests run the same way as any other unit test in a software pipeline: on every pull request, before code merges, so a leaky feature is rejected at commit time rather than discovered months later when a strategy that looked great in research fails in live trading.

Worked example. A CI test asserts that for every row in the training set, feature_timestamp <= label_timestamp. A researcher adds a new feature that computes a 30-day rolling average using .rolling(30, center=True) — a pandas default that, by centering the window, quietly pulls in 15 days of future data. The CI test fails the build immediately, flagging the exact feature and row where the timestamp check is violated, rather than letting the bug reach a live strategy that then underperforms its backtest by a wide margin.

Beyond timestamp checks, common CI leakage tests include verifying that scalers and imputers are fit only on training folds (never on the full dataset before splitting) and that any target-encoding of categorical features uses only past labels — both classic ways leakage sneaks in without an obvious symptom.