Feature Importance with SHAP

Once a model makes a prediction, you want to know why, which features drove it, and by how much. The default tree importances (Mean Decrease Impurity) are notoriously misleading: they're computed in-sample and biased toward features with many possible split points. SHAP (SHapley Additive exPlanations) fixes this with an idea borrowed from cooperative game theory: treat the features as players in a game whose "payout" is the prediction, and split that payout among them fairly.

The fairness comes from Shapley values, the unique way to divide a payout so each player gets its average marginal contribution, honoring symmetry and adding up exactly. Applied to a model, each feature's SHAP value is how much it pushed this particular prediction above or below the average prediction.

The formula and its one crucial property

A feature $i$'s SHAP value is its average marginal contribution across all the orders in which features could be added:

$$\phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!\,(|F| - |S| - 1)!}{|F|!}\,\big[f(S \cup \{i\}) - f(S)\big].$$

In words: look at every subset $S$ of the other features, see how much the prediction $f$ jumps when you add feature $i$ to it, and average those jumps with the weights shown ($F$ is the full feature set). The single most useful consequence is additivity: the SHAP values plus a base value reconstruct the prediction exactly,

$$f(x) = \phi_0 + \sum_{i} \phi_i,$$

where $\phi_0$ is the average prediction over the data. So SHAP doesn't just rank features, it accounts for a prediction, cent for cent. Sum a feature's absolute SHAP values across all predictions and you get its global importance; look at one row and you get a local explanation of that single trade.

How it compares

Property	MDI (default feature_importances_)	Permutation (MDA)	SHAP
In-sample or out-of-sample	In-sample	Out-of-sample	Either (use OOS/purged)
Local (per-prediction)	No	No	Yes
Global ranking	Yes	Yes	Yes (mean |SHAP|)
Adds up to the prediction	No	No	Yes
Biased to high-cardinality features	Yes	Less	No
Cost	Free	Moderate	Higher (exact for trees)

SHAP is the only one of the three that is additive and gives per-prediction explanations, and it avoids MDI's bias toward continuous, high-cardinality features. For tree ensembles, TreeSHAP computes the exact values efficiently.

Worked example

A gradient-boosted model outputs a probability that a trade will win. Across the dataset the average predicted probability is $\phi_0 = 0.50$, that's the base value. For one specific trade the model predicts $0.60$. TreeSHAP decomposes the $+0.10$ gap:

• momentum: $\phi = +0.12$
• volatility: $\phi = -0.05$
• bid-ask spread: $\phi = +0.03$

Check the additivity: $0.50 + 0.12 - 0.05 + 0.03 = 0.60$, exactly the prediction. The story reads cleanly: strong momentum pushed this trade's win probability up, high volatility pulled it back down, a tight spread helped a little. Now aggregate: average $|\phi|$ over all trades and suppose you get momentum $0.09$, volatility $0.06$, spread $0.02$, and a raw price-level feature $0.005$. That ranking is honest, whereas the MDI ranking on the same model put the raw price level near the top purely because its thousands of distinct values offered the most split points. SHAP demotes the artifact and promotes the feature that actually moves predictions.

Pitfalls

• Compute it out-of-sample. SHAP on the training set explains how the model fit, which on noisy data is partly memorization. For honest importance, compute SHAP on purged, embargoed validation data, see Tree Ensembles in Finance.
• Correlated features share credit. Shapley values split contribution more fairly than MDI, but when two features are highly correlated the credit is divided between them, which can make each look weaker than the pair truly is. Cluster correlated features and read importance per cluster.
• Noisy on small samples. With few effectively-independent observations (overlapping labels), SHAP rankings wobble from fold to fold. Check stability across folds before trusting a ranking.
• Local explanations are conditional on the model. They faithfully explain this model; a different, equally-valid model can attribute differently.

In interviews

If asked "how would you explain what your model learned, and why not just use feature_importances_?", say that MDI is in-sample and biased toward high-cardinality features, and reach for SHAP: Shapley values from game theory that split each prediction into additive per-feature contributions summing to the prediction, giving both local and global importance. Add the finance caveats, compute it on purged out-of-sample data, cluster correlated features, and never confuse importance with causation or a tradeable signal.