mRMR: Minimum Redundancy Maximum Relevance

Rank the top 20 features purely by relevance to the target — say, by mutual information — and you can easily end up choosing 15 different flavors of "20-day volatility" (raw, log, rank-transformed, winsorized) that all happen to score well individually, because they're all measuring roughly the same thing. Picking features one at a time by relevance alone ignores the fact that once you already have one volatility measure, adding nine more nearly-identical ones adds almost no new information despite each one individually looking useful. mRMR builds redundancy directly into the selection criterion to prevent exactly this.

The idea: relevance minus redundancy, feature by feature

mRMR builds a feature set incrementally. At each step, having already selected a set $S$, it scores every remaining candidate feature $f$ by

$$\text{score}(f) = I(f; Y) - \frac{1}{|S|}\sum_{s \in S} I(f; s)$$

and adds whichever candidate scores highest. In plain English: $I(f;Y)$ is how relevant candidate $f$ is to the target $Y$ (its mutual information, as in a simple relevance screen); the second term is the average mutual information between $f$ and every feature already chosen — how redundant $f$ would be given what's already in the set. The candidate that wins is the one offering the best relevance net of how much it overlaps with what's already there, so the selection actively seeks features that add something genuinely new rather than reinforcing the same signal.

Worked example: picking 3 features from 5 candidates

Suppose relevance scores (mutual information with forward return) for five candidates are: 20-day vol = 0.40, 60-day vol = 0.38, 5-day momentum = 0.30, 60-day momentum = 0.28, sector = 0.15. A pure relevance ranking would pick 20-day vol, 60-day vol, and 5-day momentum — but 20-day and 60-day volatility are themselves highly redundant with each other (mutual information between them, say, 0.35). mRMR's first pick is still 20-day vol (highest raw relevance, no redundancy penalty yet, since $S$ is empty). For the second pick, 60-day vol's score drops to roughly $0.38 - 0.35 = 0.03$ once its redundancy with 20-day vol is subtracted, while 5-day momentum's score stays near its full $0.30$ (low overlap with a volatility feature) — so mRMR picks 5-day momentum second instead. The third pick then weighs 60-day vol, 60-day momentum, and sector against redundancy with both already-chosen features, likely favoring 60-day momentum or sector over the now doubly-redundant 60-day vol. The resulting set of three covers volatility, short-term momentum, and a genuinely distinct third axis — a much more information-diverse set than the naive top-3-by-relevance list.

What this means in practice

mRMR is a good default when a feature library contains many engineered variants of the same underlying quantity (which is nearly always true — every volatility, momentum, or valuation concept typically gets computed at several lookback windows and transforms), because it directly targets the "many correlated near-duplicates" failure mode that simple relevance ranking is blind to. It's more expensive to run than a plain relevance screen, since the redundancy term must be recomputed against every already-selected feature at each step, and the final feature count still has to be chosen by the practitioner, usually via cross-validated model performance rather than a fixed rule.