He (Kaiming) Initialization

Xavier initialization keeps a layer's output variance equal to its input variance — but that derivation assumes the activation function is roughly symmetric around zero, like tanh. ReLU isn't: it clips every negative input to exactly zero, discarding about half the signal outright. Plug Xavier's weight scale into a ReLU network and each layer's surviving variance is roughly halved, compounding into a signal that vanishes within a dozen layers. He (Kaiming) initialization fixes this with one change: double the variance to make up for what ReLU throws away.

The analogy: a filter that blocks half the light

Imagine light passing through a chain of one-way filters, each one blocking every ray coming from below a certain angle — roughly half the incoming light, on average, for a filter facing a random scene. If you want the same brightness on the far side of ten stacked filters as you started with, each filter needs to inject roughly twice the light it's given, because half is discarded at each stage. ReLU is that filter: it passes positive values through unchanged and blocks every negative value completely. He initialization is the "inject twice as much" correction, computed exactly rather than guessed.

The formula

For a layer with $n_{in}$ inputs feeding a ReLU activation, He initialization draws weights with mean 0 and variance:

$$\text{Var}(W) = \frac{2}{n_{in}}$$

In words: this is Xavier's variance formula (which used $2/(n_{in}+n_{out})$, roughly $1/n_{in}$ for equally-sized layers) with an extra factor of 2 built in, and using only fan-in rather than the average of fan-in and fan-out — the correction is applied at the point where the signal is actually halved, going into the ReLU, not spread across both directions.

Worked example 1: comparing the two schemes at the same layer size

A layer with $n_{in} = 500$. Xavier (assuming $n_{out} = 500$ too) gives variance $2/1000 = 0.002$, standard deviation $\approx 0.045$. He gives variance $2/500 = 0.004$, standard deviation $\approx 0.063$ — about $\sqrt{2}$ times larger. That factor of $\sqrt{2}$ in standard deviation (factor of 2 in variance) is precisely what's needed to offset ReLU zeroing out half the units on average.

Worked example 2: what happens if you use Xavier weights with ReLU anyway

Take a 20-layer ReLU network where each layer has $n_{in} = 200$, using Xavier's variance $2/400 = 0.005$ instead of He's $2/200 = 0.01$. Per layer, the expected output variance multiplier is roughly $n_{in} \times \text{Var}(W) \times \tfrac{1}{2}$ (the $\tfrac12$ from ReLU zeroing half the units): with Xavier that's $200 \times 0.005 \times 0.5 = 0.5$ — variance is cut in half at every layer. After 20 layers, the surviving variance has shrunk by $0.5^{20} \approx 0.00000095$, essentially zero: the signal has vanished before training starts. With He's variance the same calculation gives $200 \times 0.01 \times 0.5 = 1.0$ per layer — variance stays flat across all 20 layers.

What this means in practice

He initialization is the default for any network using ReLU or its close relatives (Leaky ReLU, ReLU6) and is built into every major framework (nn.init.kaiming_normal_ in PyTorch, with a mode argument choosing fan-in or fan-out). It matters most in networks without normalization layers — batch norm or layer norm partially re-corrects variance drift on their own, which is why deep, unnormalized ReLU networks with the wrong initializer are the cases where this bites hardest, often showing up as a network that simply never starts learning.