Recursive Feature Elimination

Screening methods like mutual information or mRMR rank features by how they relate to the target in isolation or against a running set — but they never actually ask the model that will be used in production which features it finds useful once every other feature is already present. A feature that scores modestly on its own might turn out to be essential once combined with two others in a tree-based model's splits, or a feature that scores well alone might become nearly worthless once a better-correlated feature is already in the model. Recursive feature elimination sidesteps this by letting the actual model, trained repeatedly, do the judging.

The idea: train, rank by the model's own importance, drop the weakest, repeat

Recursive feature elimination (RFE) works as a loop:

1. Train the target model (linear coefficients, tree-based feature importances, or similar) on the current feature set.
2. Rank features by the model's own importance measure — coefficient magnitude, mean decrease in impurity, permutation importance.
3. Drop the single lowest-ranked feature (or a small bottom fraction).
4. Retrain on the reduced set and repeat, until a target feature count is reached.

Formally, if $R^{(k)}$ is the ranking of the $k$ remaining features at step $k$, the loop drops

$$f^* = \arg\min_{f \in R^{(k)}} \text{importance}(f)$$

at every iteration. In plain English: at each round, whichever feature the model currently trusts least gets removed, and the model is given a chance to re-evaluate everyone else's importance without that feature around — which matters, because a feature's apparent importance can shift substantially once a correlated competitor is gone.

Worked example: eliminating down from six to three features

Starting with six candidate features, a tree-based model's first-round importances are: momentum 0.30, volatility 0.25, size 0.20, sector 0.10, liquidity 0.09, a near-duplicate volatility variant 0.06. RFE drops the duplicate volatility variant (lowest importance). Retrained on the remaining five, volatility's importance rises to 0.32 — it absorbs the signal the duplicate used to share — while liquidity drops to 0.05 and is eliminated next. Retrained again on four features, sector's importance falls further as momentum and volatility absorb more of the explanatory weight, and sector is eliminated third. The final three-feature set — momentum, volatility, size — was arrived at by watching how importances shifted at each removal, something a single one-shot ranking of all six features from the start would never reveal, since the duplicate volatility feature initially looked mediocre but not obviously worse than liquidity or sector.

What this means in practice

RFE tends to produce feature sets tailored specifically to the model being used, since it uses that model's own importance measure — a set selected via RFE with a linear model may not transfer well to a tree-based model, so feature selection and model choice are best treated together rather than as separate pipeline stages. It's also computationally heavier than one-shot screening, since it requires retraining at every elimination step, which matters for slow-to-train models or large feature counts, and — like any selection procedure — it must be run inside a cross-validation loop rather than on the full dataset once, or the selection itself becomes a source of overfitting.