Mutual Information Feature Screening

Ordinary correlation only detects linear relationships: if a feature swings the target up when small, down when large, and back up when very large (a U-shape), the Pearson correlation between them can come out close to zero even though the feature is highly informative about the target — a screening pass based on correlation alone would throw it away. Mutual information fixes this by measuring something more fundamental: how much knowing the feature reduces your uncertainty about the target, regardless of the shape of the relationship between them.

The idea: information gain, not linear fit

Mutual information between a feature $X$ and target $Y$ is defined from entropy, the standard measure of uncertainty:

$$I(X; Y) = H(Y) - H(Y \mid X)$$

In plain English: $H(Y)$ is how uncertain you are about the target before seeing the feature at all; $H(Y \mid X)$ is how uncertain you remain about the target after learning the feature's value. Their difference is the mutual information — how much uncertainty about $Y$ the feature $X$ resolves. If knowing $X$ tells you nothing about $Y$, $I(X;Y) = 0$; the more $X$ pins down $Y$ (in any functional shape, linear or not), the larger $I(X;Y)$ grows. Screening ranks every candidate feature by its estimated mutual information with the target and keeps the top-scoring ones.

Worked example: a U-shaped relationship correlation misses

Suppose a feature — distance from a stock's price to its 52-week high — has the property that returns are elevated both very close to the high (momentum continuation) and very far below it (deep value bounce), with weak returns in between. Computed over simulated data, the Pearson correlation between this "distance" feature and forward return comes out near $\rho \approx 0.02$, essentially flagging it as useless — the U-shape means the linear fit sees no net relationship. Estimating mutual information on the same data, however, gives a value clearly above zero, because knowing where a stock sits relative to its 52-week high genuinely does shrink your uncertainty about its forward return, just not in a way a straight line can express. A correlation-only screen discards this feature; a mutual-information screen correctly keeps it.

What this means in practice

Mutual information screening is especially valuable in financial feature engineering because many real relationships genuinely are non-monotonic — mean-reversion effects that flip sign at extremes, volatility effects that matter only past a threshold, seasonal effects that aren't linear in time. In practice mutual information must be estimated from finite data (commonly via binning or k-nearest-neighbor estimators), and those estimates are noisier and more sensitive to sample size and bin choice than a simple correlation coefficient, so results should be checked for stability across different sample splits before trusting a feature ranking built on it.