Feature Selection: Filter, Wrapper, Embedded

A quant with 400 candidate signals and 3 years of daily data has a real problem: with that many features and that little data relative to them, a model can find spurious combinations that fit the past and predict nothing about the future. Feature selection cuts the set down before that happens, and the three standard families — filter, wrapper, embedded — differ in how directly they consult the model being trained, which is exactly the trade-off between speed and accounting for how features interact.

The three families

Filter methods rank each feature by a statistic computed independently of any model — correlation with the target, mutual information, an ANOVA F-score — and keep the top-scoring subset. They're cheap to compute even on thousands of features, but a feature that's useless alone and powerful in combination (two signals that only predict returns when multiplied together) gets thrown away, since filters never look at combinations.

Wrapper methods treat feature selection as a search problem over subsets, using the target model's actual validation performance as the score. Forward selection starts empty and adds the feature that helps most at each step; backward elimination starts full and removes the one that hurts least. Both directly capture interactions, because they retrain the real model on each candidate subset — but retraining hundreds of times is expensive, and testing many subsets against the same validation data reintroduces a subtler overfitting risk of its own.

Embedded methods fold selection into training. Lasso regression's L1 penalty drives some coefficients exactly to zero as part of ordinary fitting; a tree ensemble's built-in feature-importance scores fall out of how often and how effectively each feature was used to split. Both get much of a wrapper's model-awareness in a single training run rather than hundreds of them.

Worked example

Starting from 400 candidate factors, a mutual-information filter first cuts to the top 120 that individually show any relationship with forward returns — a few seconds of computation. From those 120, a Lasso regression (embedded method) is fit with a regularization strength tuned by cross-validation, and its L1 penalty drives 85 of the 120 coefficients to exactly zero, leaving 35 nonzero factors in a single training pass. A wrapper-style forward selection is then run only on those 35, since 35 is small enough to make retraining a full gradient-boosted model at each step affordable, and it settles on a final set of 18 factors whose validation performance keeps improving as each is added.

What this means in practice

Layering the three — a fast filter to cut a huge candidate set down, an embedded method to do the heavy lifting, a wrapper only on what's left — is standard practice because a wrapper search over 400 features directly would need thousands of retrains.