Partial Dependence Plots

Permutation importance tells you that a gradient-boosted tree relies heavily on "12-month momentum," but it doesn't tell you how. Is the relationship roughly linear — more momentum, proportionally more predicted return? Does it flatten out past a certain point? Does it reverse for extreme values, the model having learned that very strong momentum tends to mean-revert? A tree ensemble with hundreds of trees has no single coefficient you can point to the way a linear regression has a slope. A partial dependence plot recovers that shape directly from the trained model's behavior, without needing to understand its internals.

An analogy: tasting a stew one ingredient at a time

Imagine a complex stew with a dozen ingredients, and you want to know how the overall taste changes as you vary just the amount of salt, independent of everything else the cook happened to add each night. You can't just compare a salty night to a bland night, because those nights might also differ in every other ingredient. Instead: take the exact same pot on a given night, hypothetically re-season it with a range of salt levels while keeping every other ingredient exactly as that night's batch had it, and taste each version. Average that taste-shift across many different nights' base recipes, and you get a clean read on what salt alone does to the stew, averaged over all the ways the rest of the recipe varies in practice.

The mechanics, one step at a time

For a chosen feature $x_j$ (say, momentum) and a grid of values it might take, the partial dependence at a value $v$ is:

$$\widehat{PD}(v) = \frac{1}{n} \sum_{i=1}^{n} \hat{f}(v, x_{i,-j})$$

where $\hat{f}$ is the trained model, $x_{i,-j}$ means "observation $i$'s values for every feature except $j$," and the value $v$ is substituted in for feature $j$ on every single observation before averaging the model's predictions. In plain English: for each candidate momentum value $v$, pretend every single stock in your validation set had exactly that momentum reading (keeping every other feature — size, volatility, sector — as that stock's real values), run the model, and average the predictions. Repeat across a grid of $v$ values and plot the resulting average prediction against $v$: that curve is the partial dependence plot.

Worked example: momentum's effect, holding everything else fixed

Suppose you compute average predicted return at five momentum grid points, holding all other features at each stock's real values: at momentum $=-10\%$, the average predicted return is $-0.3\%$; at $0\%$, it's $0.1\%$; at $10\%$, it's $0.6\%$; at $20\%$, it's $0.8\%$; at $30\%$, it's $0.5\%$. The curve rises roughly linearly from $-10\%$ to $20\%$ momentum, then bends back down past $20\%$. That bend is the discovery a coefficient-based summary would have hidden entirely: the model has learned that extreme momentum readings are treated differently than moderate ones, consistent with a mean-reversion effect the model picked up in the data, and a plain "momentum matters, coefficient +0.03" summary would never have surfaced this reversal.

What this means in practice

Partial dependence plots are the standard first move for understanding a black-box model's learned relationships, and they're especially valuable for sanity-checking that a model hasn't learned something economically implausible — a monotonic momentum effect is expected, but a partial dependence curve that suddenly spikes at an obscure feature value with almost no data support is a red flag for overfitting to a thin slice of the training set. They average over the other features rather than conditioning on them, which is efficient but can mislead when features are strongly correlated, since the "hypothetical" combinations being averaged over may not resemble any real observation.