Pruning Redundant Features by Correlation
Pruning by correlation drops features that are nearly duplicates of ones already kept, since two highly correlated inputs add noise and instability to a model without adding real new information.
A dataset with both "20-day price momentum" and "21-day price momentum" as separate features isn't giving a model two pieces of information — it's giving it one idea twice, which can destabilize linear models and waste capacity in tree-based ones. Pruning by correlation removes this kind of redundancy by computing pairwise correlations across all candidate features and dropping one of any pair that's correlated above a chosen threshold, typically somewhere around 0.90 to 0.95.
When two features are highly correlated, keeping both adds redundancy and instability rather than new information, so a simple, defensible rule is to compute the full pairwise correlation matrix and drop one feature from every pair above a fixed threshold.
The usual procedure works through the correlation matrix, and when two features exceed the threshold, keeps whichever one is more interpretable, cheaper to compute, or more correlated with the prediction target, and discards the other.
Worked example. Out of 50 candidate features, a correlation matrix reveals that "trailing 12-month earnings growth" and "trailing 12-month revenue growth" have a pairwise correlation of 0.94. Since both largely capture the same "the business is expanding" signal, the analyst keeps earnings growth (closer to what the model is ultimately trying to predict) and drops revenue growth, reducing the feature set to 49 with almost no loss of information.
This kind of pruning is usually done before more sophisticated feature-selection methods, since removing near-duplicates first makes later selection steps faster and more stable.
Related concepts
Practice in interviews
Further reading
- Guyon & Elisseeff, 'An Introduction to Variable and Feature Selection' (JMLR, 2003)