ReLU, Dying Units and Leaky Variants

ReLU — output the input unchanged if it's positive, output exactly zero if it's negative — became the default activation function in deep learning because it is nearly free to compute and avoids a specific problem older activations had (vanishing gradients from saturation on both ends). But its own simplicity creates a different failure: a unit whose weights get pushed into a state where it outputs zero for every training example can become permanently stuck, unable to ever contribute to the network again.

The analogy: a light switch that can weld itself off

Think of a ReLU unit as a light switch wired so that once it's flipped off, the mechanism that would normally let you flip it back on stops working entirely — the switch has, in effect, welded itself into the off position. As long as the switch occasionally sees a positive input, it behaves like an ordinary switch: on for some inputs, off for others, learning normally. But if training ever pushes it into a state where every input in the dataset produces a negative pre-activation, the switch turns off for good, because the gradient used to update it also vanishes exactly when the input is negative — there is no signal left to teach it to turn back on.

The maths of why it gets stuck

ReLU is $f(z) = \max(0, z)$, and its derivative is $f'(z) = 1$ for $z > 0$ and $f'(z) = 0$ for $z < 0$ (undefined exactly at $z=0$, usually set to 0). During backpropagation, the gradient flowing back through a ReLU unit is multiplied by $f'(z)$:

$$\frac{\partial L}{\partial z} = \frac{\partial L}{\partial f} \cdot f'(z)$$

In plain English: if the pre-activation $z$ is negative, the local derivative is exactly zero, so no gradient at all passes back through that unit, regardless of how large the upstream error is. If a large gradient update (from a high learning rate, or an unlucky batch) pushes a unit's weights so far that $z$ is negative for every training example, then $f'(z)=0$ for all of them, no gradient can ever flow back to fix those weights, and the unit is permanently dead — this is the "dying ReLU" problem.

Leaky ReLU fixes this by never letting the gradient hit exactly zero: $f(z) = z$ for $z>0$, and $f(z) = \alpha z$ for $z<0$, with a small $\alpha$ (commonly 0.01). Its derivative is $1$ for $z>0$ and $\alpha$ for $z<0$ — small but nonzero, so a small gradient signal always survives, even when the unit is currently outputting a negative value, giving it a path back to recovery.

Worked example 1: a unit dying under a large update

A unit has weights giving pre-activation $z=0.5$ for a batch, so ReLU outputs $0.5$ and passes gradient normally. A large gradient step (say from an aggressive learning rate) drags the unit's bias down by 3.0. Now for that same batch, $z = 0.5 - 3.0 = -2.5$, ReLU outputs exactly $0$, and $f'(z)=0$: the backward pass through this unit contributes nothing to any weight update. If this holds across the entire training set (not just this batch), the unit is stuck at $z<0$ everywhere, $f'(z)=0$ everywhere, and no future gradient step can ever change its weights again — it has died.

Worked example 2: leaky ReLU recovering from the same situation

Same scenario, but with Leaky ReLU, $\alpha=0.01$. At $z=-2.5$: output is $0.01 \times -2.5 = -0.025$ (not exactly zero, but small), and the local derivative is $0.01$ (not zero). A gradient of, say, $-4.0$ arriving from upstream produces a backward contribution of $-4.0 \times 0.01 = -0.04$ — small, but nonzero. Over enough training steps, that small consistent signal can nudge the unit's bias back up until $z$ becomes positive again for some inputs, at which point the unit is fully "alive" again with derivative $1$. The 1% leak is exactly what keeps the door open.

Adjust the shape in the explorer above and notice what happens to the curve's slope near zero and in the negative region — a completely flat region, as in plain ReLU, is precisely where gradient-based learning has nothing left to push against.

What this means in practice

Dying units are more common with high learning rates, poorly scaled inputs, or bad weight initialization (see He (Kaiming) Initialization, which was specifically designed to keep ReLU pre-activations in a healthy range at the start of training). A large fraction of exactly-zero activations across a layer, checked during training, is a diagnostic sign worth watching for — it usually means the learning rate is too high or initialization is mismatched to the activation, not that Leaky ReLU is mandatory.