Clustered Feature Importance and Mean Decrease Accuracy

Suppose a model has two nearly identical momentum features — 6-month and 7-month price momentum — both genuinely useful. A standard importance ranking, like permutation importance, shuffles one feature at a time and measures how much the model's accuracy drops. But if you shuffle just the 6-month feature, the model barely notices, because the 7-month feature next to it still carries almost the same information and quietly picks up the slack. Both features get ranked as unimportant, even though momentum as a concept is one of the most important things the model uses. Correlated features hide each other's true importance.

An analogy: two witnesses telling the same story

If two witnesses independently confirm the same event, removing just one witness from the room changes nothing — the other still tells the story. A judge who concludes from this that "neither witness matters" is missing the point: together they matter a great deal, individually each looks replaceable. Feature importance among correlated features has exactly this flaw, and the fix is the same: don't judge witnesses one at a time — group them and ask what happens if the whole group is removed at once.

Mean decrease accuracy (MDA) is the base tool: for a trained model, randomly permute one feature's values (breaking its relationship to the target while leaving everything else intact) and record how much out-of-sample accuracy falls:

$$\text{MDA}_j = \text{Accuracy}_{\text{baseline}} - \text{Accuracy}_{\text{permuted } j}.$$

In plain English: a feature is "important" if scrambling it hurts the model — the bigger the drop, the more the model actually relied on that feature. Clustered feature importance first groups highly correlated features into clusters (using a correlation-based distance and hierarchical clustering), then permutes an entire cluster together rather than one feature at a time, and measures the accuracy drop for the whole group. In plain English: instead of asking "does shuffling just the 6-month momentum feature hurt," ask "does shuffling all the momentum-family features together hurt" — which correctly reveals that momentum as a family matters even though no single momentum feature looks essential on its own.

Worked example: momentum cluster versus a lone feature

Suppose a model uses six features: 1-month, 3-month, 6-month, and 12-month momentum (all pairwise correlated above 0.7), plus a standalone earnings-yield feature and a standalone size feature. Permuting 6-month momentum alone drops accuracy by only 0.3 percentage points — it looks nearly useless. Permuting all four momentum features together drops accuracy by 4.1 percentage points, revealing that the momentum family, as a group, is by far the most important input the model has, more than double the drop from permuting earnings-yield alone (1.8 points). Ranking features individually would have buried momentum near the bottom of the list and put the standalone earnings-yield feature falsely near the top — the clustered version corrects that, because it never has to face a substitute feature covering for it.

What this means in practice

Any feature-importance ranking in a financial model with fundamentals, technicals, and macro factors will contain correlated groups — value ratios often move together, as do momentum windows of different lengths — so a raw, feature-by-feature MDA ranking systematically understates the importance of anything that comes bundled with a near-duplicate. Clustering first, then permuting whole clusters, gives a ranking that better reflects which underlying ideas the model actually depends on, which is usually what a researcher wants to know before deciding what to keep, simplify, or drop.