Weight Initialization

Before a single training example is seen, every weight in a network needs some starting value, and that starting value is not a throwaway detail — it decides whether the signal passing forward through twenty layers arrives as a sensible number or as either a near-zero whisper or an overflowed mess. Get initialisation wrong and a network can fail to train at all, with a loss that never moves, long before any optimiser or learning rate is even relevant.

The analogy: passing a whisper down a long line of people

Play a game of telephone where each person, instead of just repeating what they hear, is instructed to either quietly mumble it (quieter each time) or shout it (louder each time). Mumble at every step and by the twentieth person the message has faded to silence — nobody can even guess what it was. Shout at every step and by the twentieth person it's an incomprehensible roar, clipped and distorted. Only if each person repeats the message at roughly the same volume they received it does the message survive the whole chain intact. Weight initialisation is choosing, in advance, how loudly each "person" (each layer) will repeat what it receives, so that neither shrinking nor exploding happens by the time the signal reaches the far end.

The maths: keep the variance constant across layers

If weights are initialised too small, each layer shrinks the signal's spread (variance) a little; over many layers that shrinkage compounds multiplicatively toward zero. Too large, and it compounds toward infinity. Xavier (Glorot) initialisation sets each weight by drawing from a distribution with variance chosen so this doesn't happen:

$$\text{Var}(w) = \frac{2}{n_{\text{in}} + n_{\text{out}}}$$

Here $n_{\text{in}}$ is the number of inputs feeding into a layer and $n_{\text{out}}$ the number of outputs it produces. In words: the more inputs a neuron sums together, the smaller each individual weight needs to be, so that the total spread of the summed output stays roughly the same as the spread of a single input — this cancels out the natural tendency of summing many things to inflate variance. For ReLU networks specifically, He initialisation adjusts the constant because ReLU zeroes out about half its inputs:

$$\text{Var}(w) = \frac{2}{n_{\text{in}}}$$

In words: since roughly half the neurons are "dead" (outputting zero) after a ReLU, the surviving half need proportionally more variance each to keep the overall signal strength constant — a correction Xavier's original derivation, built around sigmoid/tanh, doesn't include.

Worked example 1: signal variance across 10 layers, done two ways

Suppose each layer has 100 inputs, and weights are initialised with a fixed variance of $0.1$ regardless of layer size (a naive, size-agnostic choice) versus Xavier's $\text{Var}(w) = 2/(100+100) = 0.01$. With the naive $0.1$, if the layer sums 100 roughly-independent weighted inputs, the output variance multiplies by approximately $100 \times 0.1 = 10$ each layer. After 10 layers, that's $10^{10}$ — an explosion many orders of magnitude past anything a float can meaningfully use, saturating every downstream activation. With Xavier's $0.01$, the output variance multiplies by roughly $100 \times 0.01 = 1$ each layer — unchanged after 10 layers, exactly the stable outcome intended.

Worked example 2: why ReLU needs its own correction

A ReLU layer with $n_{\text{in}}=64$, $n_{\text{out}}=256$. Xavier's formula gives $\text{Var}(w) = 2/(64+256) = 0.00625$. He's formula, using fan-in only, gives $\text{Var}(w) = 2/64 = 0.03125$ — five times larger. Trace why: ReLU sets roughly half its outputs to exactly zero, so a layer's output variance is only about half of what the same weights would produce with a linear activation. Xavier's derivation assumes nothing gets zeroed out, so passing its (too small) weights through a ReLU layer leaves the surviving signal under-scaled, and that shortfall compounds over many layers into the same slow shrink as the naive case in worked example 1, just milder. He's larger variance is precisely calibrated to cancel the roughly 2x loss ReLU's zeroing causes, keeping output variance stable layer after layer.

Each layer's weight matrix stretches its input the way the explorer above stretches a circle into an ellipse — initialise the stretch factor too large or too small and repeated applications across many layers compound the distortion exponentially, exactly as dragging the transform's scale repeatedly would.

A network initialised too large or too small starts training from a badly-scaled point on the loss surface shown in the explorer above — even before any step is taken, a bad starting scale can put early gradients in a flat or wildly steep region that makes the first several steps unproductive or unstable.

What this means in practice

Modern deep learning libraries default to Xavier or He initialisation depending on the activation function used, and matching the initialisation scheme to the activation (He for ReLU-family, Xavier for sigmoid/tanh) is a real, not cosmetic, choice. Getting this wrong is a classic cause of a network whose loss simply never moves during the first several thousand steps — not because the optimiser or learning rate is wrong, but because the signal was already dead or saturated before training began.