Vanishing and Exploding Gradients

Backpropagation computes each layer's gradient by multiplying the gradient from the layer after it by that layer's own local slope, chained all the way back to the first layer. That chain-of-multiplications structure is efficient, but it has a dangerous property: multiply enough numbers together, and the result depends far more on whether they're typically above or below 1 than on any single one of them. A twenty-layer network where each layer's local slope is a modest $0.5$ ends up with a gradient at layer one that's been multiplied by $0.5$ twenty times over — a number so small it might as well be zero.

The analogy: a message passed through many quiet or loud rooms

Send a message down a chain of twenty rooms, where in each room the volume either drops by half or doubles before being passed to the next room. If every room halves the volume, by the twentieth room the message has been divided by $2^{20}$, over a million — completely inaudible, gone, regardless of how loud it started. If every room instead doubles it, by the twentieth room it's been multiplied by over a million — a deafening, meaningless roar. Only if the volume stays roughly constant, neither systematically halving nor doubling, room after room, does the message survive the whole chain in a usable form. Gradients passing backward through a deep network face exactly this fate, room being layer, volume being gradient magnitude.

The maths: a product of many local slopes

For a gradient to travel from the loss back to layer 1 through $L$ layers, backpropagation multiplies $L$ local slope terms together (as in the chain rule underlying backpropagation):

$$\frac{\partial L}{\partial \theta_1} = \frac{\partial L}{\partial a_L}\prod_{\ell=1}^{L-1} \frac{\partial a_{\ell+1}}{\partial a_\ell}$$

In words: the gradient reaching the earliest layer is the product of every layer's local sensitivity, one after another. If each term in that product is, on average, some constant $c$, the whole product behaves roughly like $c^{L}$ — and small deviations of $c$ from exactly $1$ get raised to a large power. $c<1$ drives the product toward $0$ (vanishing); $c>1$ drives it toward infinity (exploding). This is why sigmoid activations, whose slope is at most $0.25$ and typically much less, are especially prone to vanishing gradients in deep networks, while poorly scaled weight initialisation (large weights multiplying at every layer) is a common cause of exploding gradients.

Worked example 1: vanishing, twenty layers of sigmoid

Suppose each layer's local slope (a combination of the weight and the sigmoid's derivative) averages $c=0.6$. After 5 layers: $0.6^5 \approx 0.0778$. After 10 layers: $0.6^{10}\approx 0.0060$. After 20 layers: $0.6^{20}\approx 0.0000366$. A gradient of magnitude $1$ at the output has shrunk to about $0.0000366$ by layer 20 — for any reasonable learning rate, the update to layer 20's weights is now so tiny it's effectively zero, and that layer stops learning in any practical sense, even though nothing is technically broken.

Worked example 2: exploding, the other direction

Same setup but $c=1.5$ instead. After 5 layers: $1.5^5\approx7.59$. After 10 layers: $1.5^{10}\approx57.7$. After 20 layers: $1.5^{20}\approx3325$. A gradient of magnitude $1$ has grown to over three thousand by layer 20 — multiplied by a learning rate of, say, $0.01$, that's still a step of size $33$, likely large enough to send the corresponding weight to a wildly different value in one step, overshooting any minimum and potentially producing numerical overflow (NaN losses) if the growth continues over more layers or more training steps.

The explorer above shows overshooting when a step is too large for the local curvature — an exploding gradient produces exactly this failure mode, but the "step too large" comes from the gradient itself blowing up rather than from a badly chosen learning rate.

The curve here traces exponential growth or decay — precisely the shape of $c^L$ from the maths above, and the reason a small per-layer imbalance in slope becomes a catastrophic end-to-end effect once raised to the power of many layers.

What this means in practice

Vanishing gradients show up as early layers of a deep network that simply never seem to learn, no matter how long training runs, while later layers train fine — the loss moves, just not from where it needs to. Exploding gradients show up more dramatically, often as a loss that suddenly spikes to NaN partway through training. The standard fixes attack the same root cause from different angles: careful weight initialisation (keeping each layer's slope near 1 from the start), ReLU-family activations (avoiding sigmoid's inherently small maximum slope), residual/skip connections (giving gradients a path that bypasses some of the multiplication), normalisation layers, and gradient clipping (a hard cap on gradient magnitude, a direct fix for exploding specifically).