Integrated Gradients

Explaining a network's prediction by looking at the gradient at the actual input point can be badly misleading whenever the network has saturated — the gradient at a specific point tells you how the output would change with an infinitesimal nudge right there, but a feature that saturated a neuron long before reaching its current value can show a near-zero gradient even though changing that feature substantially would clearly change the output. Integrated gradients fixes this by not trusting a single snapshot at all — it walks a straight path from a neutral baseline input to the actual input, and accumulates sensitivity all along the way.

The analogy: crediting the whole climb, not just the last step

Imagine measuring how much a hiker's satisfaction improved during a climb by only checking their rate of enjoyment gain at the exact summit — if they had already plateaued in mood just before reaching the top (the last few steps added almost nothing new), that final measurement would suggest the whole climb barely mattered, even though the climb from the trailhead clearly did. A fairer measure adds up how much satisfaction changed at every single step of the climb, from the trailhead (a neutral starting point) all the way to the summit (the actual endpoint). Integrated gradients does exactly this for a feature: instead of reading the gradient only at the actual input, it sums the gradient along the entire straight-line path from a baseline (often all zeros, or an average input) to the input in question.

The formula, piece by piece

$$\text{IG}_i(x) = (x_i - x_i') \int_0^1 \frac{\partial f\big(x' + t(x-x')\big)}{\partial x_i}\,dt$$

Here $x_i'$ is the baseline value of feature $i$ (the neutral starting point), and $x_i - x_i'$ is the total distance travelled for that feature. The integral walks a parameter $t$ from 0 (at the baseline) to 1 (at the actual input), evaluating the gradient of the model's output with respect to feature $i$ at every intermediate point along that straight line, and averaging those gradients. In plain English: rather than trusting the gradient's value only right at the destination, integrated gradients checks the gradient at many points along the entire journey there and adds up the contribution earned at each step — so a feature whose gradient was large early in the journey but flattened out near the end still gets proper credit.

Worked example 1: verifying completeness on $f(x)=x^2$

Take $f(x)=x^2$, baseline $x'=0$, input $x=3$. The exact integral: $\text{IG} = (3-0)\int_0^1 2(0+t\cdot3)\,dt = 3\int_0^1 6t\,dt = 3\times3 = 9$. Approximating with 4 discrete steps at $t=0.25,0.5,0.75,1$: gradients are $2(0.75)=1.5$, $2(1.5)=3.0$, $2(2.25)=4.5$, $2(3)=6.0$; averaging gives $(1.5+3.0+4.5+6.0)/4=3.75$, times $(x-x')=3$ gives $11.25$ — already in the right neighborhood with only 4 steps, converging to the exact value 9 as more steps are used. Crucially, $9 = f(3)-f(0) = 9-0$ exactly — this "completeness" property (attributions sum to the actual output difference) is guaranteed by construction, never approximately true by luck.

Worked example 2: catching a saturated feature saliency misses

A sigmoid-based network is deeply saturated at the actual input, giving a raw gradient there of essentially $0.0004$ — plain saliency would report this feature as nearly irrelevant. Walking the path from baseline $x'=0$ to $x=8$ in 4 steps at $x=2,4,6,8$, the gradients (of a logistic function $\sigma(x)$) are roughly $0.10, 0.018, 0.0025, 0.0004$ — large near the start of the path, where the sigmoid was still in its steep region, and small only near the end. Averaging these four gradients, $(0.10+0.018+0.0025+0.0004)/4 \approx 0.030$, times the path length $8$, gives an integrated gradient of about $0.24$ — a meaningfully nonzero attribution that the single end-point gradient completely missed.

Drag the input here and watch the slope flatten near the extremes — a raw gradient taken only at a saturated point looks tiny, but integrated gradients credits the steep slope the path passed through earlier.

What this means in practice

Integrated gradients is the standard choice for explaining deep networks with smooth, differentiable architectures — image classifiers, embedding-based ranking models, deep trading signal networks — anywhere plain saliency risks reporting "no importance" for features the model actually relies on heavily but happens to be locally saturated with respect to.