Selecting Features Inside the CV Loop

Suppose you have 500 candidate features and you keep the 20 with the highest correlation to next-day return, computed on your entire dataset. Then you split into train and test folds, fit a model on the 20 selected features, and evaluate on the test fold. The test-fold performance will look good — and it will be a lie. The selection step already looked at the test fold's outcomes when it picked those 20 features, so the test fold is no longer a fair, unseen check. This is one of the most common and most invisible sources of inflated backtest performance in quantitative research.

Why it leaks

With enough candidate features, some fraction will correlate with the target purely by chance, especially in noisy financial data with limited history. Ranking all 500 features by their correlation to the full dataset's outcomes and keeping the top 20 is itself a fitting step — it uses the labels. If that ranking is done once on the whole dataset before splitting, the "test" fold's own outcomes contributed to which features survived, so the model is evaluated on data it was indirectly fit to. The test score no longer measures out-of-sample skill; it measures how well 20 features chosen partly because they fit those same test rows continue to fit those rows.

The fix: selection lives inside the fold

Feature selection must be repeated independently inside every training fold, using only that fold's training data, exactly like fitting the model itself. For each fold: select features using only the training rows of that fold, fit the model using only those selected features and those training rows, then evaluate on the held-out rows — which never influenced either the selection or the fit.

right: select inside each fold train fold test fold select + fit on train only evaluate on untouched test fold

Worked example

Two researchers each test 200 candidate momentum-style features against 3 years of daily returns for one stock — a small sample relative to the number of candidates. Researcher A ranks all 200 by correlation with the full-sample return, keeps the top 10, then does 5-fold cross-validation with those 10 fixed features: the reported out-of-sample Sharpe ratio comes out at 1.8. Researcher B repeats feature selection inside each of the 5 folds, using only that fold's training data each time, and the different folds keep noticeably different top-10 lists, several overlapping only partially. Her honestly out-of-sample Sharpe ratio comes out at 0.4. The gap between 1.8 and 0.4 is not a difference in modelling skill — it is the size of the leak in Researcher A's process, and it is the number that would have shown up as a live-trading disappointment.

Related traps

This is the same discipline that Nested Cross-Validation formalises for hyperparameter tuning: an inner loop does the fitting decision, an outer loop provides the honest evaluation, and the two never share data. It also compounds with high-cardinality feature sets from One-Hot Encoding and the High-Cardinality Problem, where target encoding computed outside the fold loop is exactly this same bug wearing a different name.

What is safe to do before the split

Not every preprocessing step needs to live inside the fold. Steps that use only the feature data and never touch the label — dropping a column that is constant across the entire dataset, converting a timestamp into a day-of-week indicator, deduplicating rows — do not leak, because there is no label information for them to smuggle across the train/test boundary. The dividing line is not "does this happen before or after the split" in general; it is specifically whether the step's decision depended, even indirectly, on the outcome variable. A useful habit is to write down, for every preprocessing step in a pipeline, whether it is a function of $X$ alone or a function of $(X, y)$ together — anything in the second category belongs inside the fold.

The computational cost, and why it is worth paying

Redoing feature selection inside every fold multiplies the computational cost by the number of folds, which can matter when selection itself is expensive — a wrapper method that refits a full model for every candidate feature subset, for instance. This cost is not optional overhead to be economized away; it is the price of an honest number. A cheaper, fully out-of-fold-selected pipeline that reports a more modest but real Sharpe ratio is worth more than a cheaper, leaked pipeline reporting an impressive one, and the gap between the two — as the two-researcher example shows — is often the entire difference between a strategy that survives contact with live markets and one that does not.