Mutual Information as a Dependence Measure

You're screening candidate features for a signal and computing correlation with next-day return for each one. A feature has a correlation of 0.02 with the target — essentially useless by that measure — and you discard it. But suppose the true relationship is that the feature strongly predicts the magnitude of the next move (big feature value in either direction means a big move, in either direction) with no consistent sign — a U-shaped or symmetric nonlinear relationship. Correlation, which only measures straight-line co-movement, is blind to this entirely; it can report exactly zero for a feature that's actually a near-perfect predictor of volatility. You need a measure of dependence that doesn't assume the relationship is a straight line.

An analogy: guessing a friend's mood from the weather

Suppose you know almost nothing about how weather affects your friend's mood, and you want to measure how much knowing today's weather narrows down your guess about their mood. If weather and mood were completely unrelated, learning it's sunny wouldn't shift your guess about their mood at all — you'd still be spread across "happy," "grumpy," "tired" with the same uncertainty as before. But if sunny days reliably correlate with a happier mood and stormy days reliably correlate with either extreme (energized by a big project, or grumpy about the commute) — a relationship correlation would badly undersell because it's not a simple "up weather, up mood" line — mutual information still detects it, because it just asks: does knowing the weather shrink your uncertainty about mood at all, in any pattern, not specifically a linear one?

The measure, one symbol at a time

Recall entropy, $H(Y)$, measures how uncertain you are about a random variable $Y$ before learning anything else — high entropy means $Y$ could be almost anything with similar odds; low entropy means $Y$ is fairly predictable already. Conditional entropy, $H(Y \mid X)$, measures how uncertain you remain about $Y$ after learning $X$. Mutual information is defined as the reduction in uncertainty:

$$I(X;Y) = H(Y) - H(Y \mid X) .$$

In plain English: mutual information is exactly how much learning $X$ shrinks your uncertainty about $Y$ — if knowing $X$ tells you nothing new about $Y$, $H(Y\mid X) = H(Y)$ and $I(X;Y)=0$; if knowing $X$ pins $Y$ down completely, $H(Y\mid X)=0$ and $I(X;Y)=H(Y)$, its maximum possible value. Unlike correlation, this definition never assumes the relationship between $X$ and $Y$ has any particular shape — it works the same whether the dependence is linear, curved, cyclical, or a relationship in variance rather than level.

An equivalent, more computable form for discrete or binned variables:

$$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)} .$$

In plain English: for every combination of $X$-value and $Y$-value, compare how often that combination actually occurs, $p(x,y)$, to how often it would occur if $X$ and $Y$ were completely independent, $p(x)p(y)$; if the actual joint frequency consistently differs from the independent prediction, $I(X;Y)$ is large — this is the same logic as a chi-squared independence test, but expressed in information units (bits, if using $\log_2$) rather than a test statistic.

Worked example 1: a U-shaped relationship correlation misses

Feature $X \in \{-2, -1, 0, 1, 2\}$, target $Y = X^2 \in \{4, 1, 0, 1, 4\}$, each value equally likely ($p=0.2$). The sample correlation is exactly zero — $X$ has mean 0 and is symmetric, so positive and negative $X$ values contribute canceling terms to the covariance:

$$\text{Cov}(X,Y) = \frac{1}{5}\sum_i (x_i - 0)(y_i - \bar y) = \frac{1}{5}\big[(-2)(4-2) + (-1)(1-2) + 0 + (1)(1-2) + (2)(4-2)\big] = \frac{1}{5}[-4+1+0-1+4] = 0 .$$

Yet $Y$ is a deterministic function of $X$ — knowing $X$ tells you $Y$ exactly, with zero remaining uncertainty. Since $H(Y \mid X) = 0$ (no uncertainty left once $X$ is known),

$$I(X;Y) = H(Y) - 0 = H(Y) .$$

With $Y$ taking values $\{4,1,0,1,4\}$, i.e., $p(0)=0.2$, $p(1)=0.4$, $p(4)=0.4$: $H(Y) = -0.2\log_2 0.2 - 0.4\log_2 0.4 - 0.4\log_2 0.4 \approx 0.464 + 0.529 + 0.529 = 1.522$ bits. So $I(X;Y) \approx 1.52$ bits — a substantial, correctly detected dependence — while correlation reported exactly 0. This is precisely the U-shaped case a feature screen based on correlation alone would silently throw away.

Worked example 2: mutual information on a coarse feature screen

Bin a candidate feature into "low / medium / high" and next-day return into "down / flat / up," and tabulate observed frequencies from 900 days:

	Return down	Return flat	Return up	Row total
Feature low	150	50	100	300
Feature medium	100	100	100	300
Feature high	50	50	200	300

If feature and return were independent, each cell should be (row total × column total)/900; e.g., for "low, down": $300 \times 300 / 900 = 100$, versus the observed 150 — well above what independence predicts. Computing $p(x,y)\log\frac{p(x,y)}{p(x)p(y)}$ for every cell and summing (using $p(\text{low})=p(\text{med})=p(\text{high})=1/3$ and $p(\text{down})=300/900$, $p(\text{flat})=200/900$, $p(\text{up})=400/900$) yields a positive $I(X;Y)$ of roughly 0.05–0.08 bits — small but clearly nonzero, correctly flagging that "high" feature values are associated with "up" days far more often than chance, a pattern that would also show up as nonzero correlation here, but mutual information reports it without ever assuming the relationship was linear in the first place, and generalizes immediately to features where it isn't.

What this means in practice

Mutual information is a standard tool in feature screening pipelines precisely because correlation systematically misses non-monotonic relationships — volatility-predicting features, regime-dependent effects, and anything U-shaped or cyclical — that are common in market data. It's used alongside correlation, not as a replacement: correlation is cheap, interpretable, and tells you the sign of a linear relationship, which mutual information does not (it's always non-negative and tells you only how much dependence exists, not its direction or shape). In practice it requires binning or density estimation for continuous variables, which introduces its own choices (bin width interacts with the same bias-variance trade-off as kernel density estimation), and it needs meaningfully more data than correlation to estimate reliably, especially with many bins.