Correlation Filtering and Redundant Features

A feature set with both "20-day realized volatility" and "20-day standard deviation of returns" is carrying the same information twice under two names. That kind of redundancy doesn't necessarily hurt a model's raw predictive accuracy much, but it does inflate the feature count for no benefit, makes coefficients in linear models unstable and hard to interpret (each of two near-identical features can absorb an arbitrary, unstable share of the same underlying effect), and slows down tree-based models by giving them redundant splits to consider. Correlation filtering is the simplest tool for cutting that redundancy down before it causes problems.

The idea: drop one of every highly correlated pair

Compute the pairwise Pearson correlation between every pair of numeric features,

$$\rho_{X,Y} = \frac{\operatorname{Cov}(X, Y)}{\sigma_X \sigma_Y} ,$$

then, for any pair whose absolute correlation exceeds a chosen threshold (commonly 0.90 or 0.95), drop one of the two — usually the one that's less interpretable, more expensive to compute, or has lower correlation with the target itself. In plain English: $\rho$ measures how tightly two features move together on a scale from $-1$ (perfectly opposite) to $+1$ (perfectly together); a value near $\pm 1$ means the two features are carrying almost the same information, so keeping both adds width to the feature matrix without adding much new signal.

Worked example: filtering a five-feature panel

A candidate feature set has: 20-day volatility, 20-day return standard deviation, 5-day momentum, 60-day momentum, and market cap. Computing the pairwise correlation matrix shows $\rho(\text{20d vol}, \text{20d std dev}) = 0.98$ — nearly identical by construction, since both measure dispersion of the same return series over the same window — while $\rho(\text{5d mom}, \text{60d mom}) = 0.35$, a much weaker relationship since they capture momentum at genuinely different horizons. Applying a 0.90 threshold flags only the vol/std-dev pair; the modeler keeps 20-day volatility (marginally more standard in practice) and drops the standard-deviation duplicate, leaving four features instead of five with essentially no loss of information.

You can explore this directly.

Drag the correlation slider toward 1 and watch the points collapse onto a single line — that's the visual signature of two features so redundant that one can be dropped with almost no information loss; drag it toward 0 and the cloud spreads out, showing two features that each contribute something the other doesn't.

What this means in practice

Correlation filtering is a cheap, fast first pass before more expensive feature selection methods — it only catches pairwise linear redundancy, though, so it can miss cases where three or more features are jointly redundant without any single pair crossing the threshold, and it says nothing about relevance to the target, only about redundancy between features. It's typically run before, not instead of, target-aware methods like mutual information screening or mRMR.