KernelSHAP

Splitting credit for a group project's outcome fairly among teammates, in the game-theoretic sense, requires checking every possible sub-team — with or without each person — and averaging how much that person's presence changed the result across all of them. For a model with $M$ features, "every possible sub-team" means $2^M$ subsets. With even 20 features that is over a million subsets per prediction explained — computationally impossible to do exactly for most real models. KernelSHAP gets a very good approximation to those exact Shapley values without ever enumerating all of them.

The analogy: crediting teammates without checking every sub-team

Imagine estimating each teammate's contribution to a project's success not by exhaustively trying every possible sub-team combination, but by sampling a representative handful of combinations — some with almost everyone present, some with almost no one, most somewhere in between — and then solving for the fairest assignment of credit consistent with the outcomes you actually observed. KernelSHAP does exactly this: it samples "coalitions" (subsets of features either present or replaced by a baseline), observes what the model predicts for each, and fits a simple weighted linear regression whose coefficients converge to the exact Shapley values as more coalitions are sampled.

The mechanics

For a coalition $z$ (a binary mask of which of the $M$ features are "present," the rest replaced by baseline values), KernelSHAP evaluates the model's prediction $f(z)$ and weights that observation using the Shapley kernel:

$$\pi(z) = \frac{M-1}{\binom{M}{|z|}\,|z|\,(M-|z|)}$$

where $|z|$ is the number of present features in that coalition. In words: this weighting formula deliberately upweights coalitions that are almost empty or almost full (which are most informative about a single feature's marginal effect) and downweights coalitions near half-full (which are numerous but less informative per-sample). Fitting a weighted least-squares regression of the sampled $f(z)$ values on the coalition masks, using these weights, produces coefficients that are the Shapley value approximations — one per feature.

Worked example 1: exact Shapley values for a simple linear model

For a genuinely linear model $f(x) = 2x_1 + 3x_2 - x_3$ with baseline $x=(0,0,0)$ and instance $x=(1,1,1)$, Shapley values have a known closed form for linear models: each feature's contribution is simply its coefficient times its own deviation from baseline, because there is no interaction to split up. That gives $\phi_1 = 2(1)=2$, $\phi_2=3(1)=3$, $\phi_3=-1(1)=-1$, summing to $4 = f(1,1,1) - f(0,0,0) = 4-0$. KernelSHAP, run on this same model with enough sampled coalitions, recovers these same three numbers — a useful sanity check, since the exact answer is known here without needing any sampling at all.

Worked example 2: the kernel weight in practice

For $M=3$ features, consider a coalition with $|z|=1$ feature present. $\binom{3}{1}=3$, so $\pi(z) = \frac{2}{3\times1\times2}=0.333$. For $|z|=2$: $\binom{3}{2}=3$, so $\pi(z)=\frac{2}{3\times2\times1}=0.333$ — same weight, by the kernel's built-in symmetry between "mostly absent" and "mostly present" coalitions. Compare a coalition with $|z|$ near $M/2$ in a larger example with $M=10$, $|z|=5$: $\binom{10}{5}=252$, giving $\pi(z) = \frac{9}{252\times5\times5}=0.0014$ — over 200 times smaller per-coalition weight than the extreme cases, exactly the "upweight the edges, downweight the middle" behavior the kernel formula is designed to produce.

The final step of KernelSHAP is literally this: a (weighted) regression fit, but on sampled coalition masks and their model outputs rather than raw features and outcomes.

What this means in practice

KernelSHAP works with any model as a black box — it never looks inside, only at input/output pairs — making it the standard choice for explaining ensembles, neural networks, or any model without a faster specialized SHAP variant like TreeSHAP, at the cost of needing many model evaluations per explained prediction and being sensitive to the choice of baseline used to represent "absent" features.