Feature Selection Bias
Choosing which features to keep in a model using the same data later used to test it inflates the model's apparent performance, because the selection process itself has already fit to that data's noise.
Prerequisites: In-Sample and Out-of-Sample Protocol
Building a predictive model usually starts with more candidate features than you'll actually use — dozens of possible signals, ratios, or indicators, most of which turn out to be useless or redundant. Feature selection is the process of narrowing that list down. The trap is doing the narrowing using the same data that will later be used to evaluate the model: if you scan 50 candidate features, keep the 10 that correlate best with returns in your sample, and then report the model's in-sample performance using those 10, you've measured a model that was built specifically to fit that sample's noise, not a model that would perform the same way on new data.
Why selection itself is a form of fitting
Every feature has some correlation with the target purely by chance, even a feature with zero real predictive power. With enough candidate features, some of them will show a strong-looking correlation in any finite sample just from noise — this is the same logic behind multiple testing. Selecting features by "keep whatever correlates best in this sample" systematically picks out the features whose noise happened to align favorably with this particular sample's outcomes. The resulting model looks great on the data it was selected against, but that apparent strength is partly an artifact of the selection process itself, and it typically doesn't survive contact with fresh data.
Worked example: 40 candidate features, 5 chosen, one held out
A researcher has 40 candidate technical indicators and wants to build a five-feature model predicting next-week returns. Using three years of data, they rank all 40 by correlation with the target and keep the top 5, which together give an in-sample R² of 0.18. Testing that same five-feature model on a fourth year of data — never touched during selection — the R² drops to 0.04. The gap isn't because the model changed; it's because the top-5 selection process, run entirely on the first three years, specifically found the five indicators whose noise best matched that period's actual outcomes. A feature selection process that instead used cross-validation during the selection step — checking each candidate feature's contribution on data it wasn't selected against — would report a lower but more honest in-sample number, closer to what the fourth year actually showed.
What this means in practice
The fix is to treat feature selection as part of the model, not a preprocessing step done before validation starts: any cross-validation or holdout evaluation must repeat the entire selection process — scanning all candidates, picking the top few — inside each fold, rather than selecting once on all the data and only cross-validating what happens afterward. With a large candidate pool relative to the sample size, expect a meaningful in-sample-to-out-of-sample performance gap even with a properly nested procedure; the honest number is always the one measured on data the selection step never saw.
Feature selection performed on the same data used to evaluate a model overstates its performance, because the selection step has already exploited that sample's noise. Selection must be re-run inside each cross-validation fold, not done once beforehand, for validation numbers to be trustworthy.
Running feature selection on the full dataset and only cross-validating the resulting fixed feature set afterward looks like proper validation but isn't — the selection step itself already leaked information from every fold into the model, since it saw the whole dataset before any fold was held out.
Further reading
- Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, ch. 7