Saliency Maps and Raw Gradient Attribution

The cheapest way to ask "which input features does this network's prediction actually depend on" is to compute the gradient of the output with respect to each input feature, exactly where the input currently sits. That single number per feature — how much the output would move for a tiny nudge in that direction — is a saliency map. It costs one backward pass, reusing exactly the machinery already built for training, but it is only ever a local, single-point reading, and that limitation is easy to forget.

The analogy: a compass reading, valid only where you stand

A compass tells you which direction is "more uphill" from exactly where you are standing — completely accurate at that spot, but it says nothing about the terrain a mile away, and it can mislead you if you happen to be standing on a small flat ledge partway up an otherwise steep mountain: the compass reads "flat, no direction matters here" even though the mountain clearly has a dominant slope overall. A raw gradient is exactly this compass reading taken at the network's actual input — accurate about the local slope of the output with respect to each feature, right at that one point, but blind to whether that point happens to sit on a locally flat plateau of an otherwise steep, important function.

The computation

For a model output $f(x)$ and input features $x_1, \ldots, x_M$, the saliency attribution for feature $i$ is simply:

$$S_i(x) = \frac{\partial f(x)}{\partial x_i}$$

computed at the specific input $x$ being explained, via one ordinary backward pass (automatic differentiation, the same operation used every training step). In words: this number says how fast the model's output would change per unit nudge to feature $i$, evaluated only at this exact input — nothing is walked, sampled, or averaged, it is a single read of the local slope. A common variant, gradient × input, multiplies $S_i(x)$ by $x_i$ itself, which better reflects total contribution when features are on very different scales.

Worked example 1: a simple linear-plus-ReLU network

Let $f(x) = \max(0,\, 2x_1 - x_2 + 3)$, evaluated at $x_1=1, x_2=0$, so the pre-activation value is $2(1)-0+3=5 > 0$ — the ReLU is active (in its linear region). The gradient there is simply the linear coefficients: $\partial f/\partial x_1 = 2$, $\partial f/\partial x_2 = -1$. Gradient×input gives $2\times1=2$ for feature 1 and $-1\times0=0$ for feature 2 — feature 2's raw gradient says "moving it would matter (coefficient $-1$)," but its gradient×input attribution says "at this specific value of zero, it isn't currently contributing anything," a genuinely different and equally valid question.

Worked example 2: where raw gradient goes wrong

Now evaluate the same network at $x_1=-5, x_2=0$: the pre-activation is $2(-5)-0+3=-7 < 0$, so the ReLU is in its flat, inactive region and outputs exactly $0$. The gradient there is $0$ for both features — raw saliency reports both $x_1$ and $x_2$ as completely unimportant at this point. But $x_1$ clearly matters a great deal to this model overall: moving it from $-5$ to, say, $1$ would flip the ReLU active and change the output substantially. The single-point gradient, sitting on this flat region, is technically correct about the local slope and yet completely misses the feature's real importance — exactly the plateau problem the compass analogy describes.

The flat tails of this curve are where raw gradient attribution goes quiet even though the input clearly still matters to the function's overall shape — the same failure mode a saturated ReLU or sigmoid layer produces inside a real network.

What this means in practice

Saliency maps are the fastest, most accessible explanation tool for any differentiable model, and remain useful as a quick first pass — spotting obviously irrelevant input regions in an image classifier, or obviously dead features in a trading signal network — but they should not be the final word on feature importance for any decision with real stakes, precisely because of the plateau blind spot.