Xavier/Glorot Initialization

Before a network sees a single training example, someone has to fill its weight matrices with starting numbers. Pick them badly and the network is broken before training begins: signals passed forward through many layers either shrink toward zero or grow without bound, and gradients passed backward do the same, making early learning nearly impossible. Xavier (also called Glorot) initialization is a specific recipe for the variance of those starting numbers that keeps signals roughly the same size from layer to layer.

The analogy: a chain of amplifiers

Picture a signal passing through a chain of audio amplifiers, one per layer of the network. If each amplifier boosts the signal even slightly too much, by the tenth stage the sound is a deafening wall of noise. If each one attenuates it even slightly too much, by the tenth stage there's nothing left to hear. To keep the signal usable at every stage, each amplifier's gain has to be tuned to the number of inputs feeding it — a stage summing many inputs needs a smaller gain per input than one summing few, or the combined loudness still drifts. Xavier initialization is that gain-tuning rule, applied to the random weights of a neural network layer.

The formula

For a layer with $n_{in}$ inputs and $n_{out}$ outputs, Xavier initialization draws each weight from a distribution with mean 0 and variance:

$$\text{Var}(W) = \frac{2}{n_{in} + n_{out}}$$

In words: the spread of the random starting weights should shrink as the layer gets wider, using the average of the fan-in and fan-out counts, so that the variance of the layer's output roughly equals the variance of its input — signal doesn't get amplified or damped, forward or backward, purely because of layer size.

Worked example 1: two layer sizes, two very different weight scales

Layer A has $n_{in} = 4$, $n_{out} = 4$: variance $= 2/8 = 0.25$, standard deviation $= 0.5$ — fairly wide weights are fine because so few inputs are being summed. Layer B has $n_{in} = 1000$, $n_{out} = 1000$: variance $= 2/2000 = 0.001$, standard deviation $\approx 0.032$ — each individual weight must be tiny, because summing a thousand of them (each roughly independent) still needs to land near the same overall scale as layer A's output. Using layer B's weight scale on layer A, or vice versa, would leave one layer's outputs far too small or far too large.

Worked example 2: why variance compounds across layers

Suppose a network has 10 stacked linear layers, each with $n_{in} = n_{out} = 100$, and each weight were instead drawn with a fixed standard deviation of $0.1$ regardless of layer size (variance $0.01$, versus Xavier's $2/200 = 0.01$ here — coincidentally close, so pick a worse example: weights with variance $0.05$, five times too large). Each layer roughly multiplies the running variance by $n_{in} \times \text{Var}(W) = 100 \times 0.05 = 5$. After 10 layers the variance has grown by a factor of $5^{10} \approx 9.8$ million — outputs have exploded into meaninglessly large numbers, and gradients flowing back through the same layers explode identically. Xavier's scaling is chosen precisely so that multiplier is close to 1 per layer, keeping the compounding factor near $1^{10} = 1$ instead.

The explorer below draws random weights from a normal distribution — set the standard deviation to $0.5$ for the small layer and to $0.032$ for the thousand-wide layer from worked example 1, and compare how tightly each is bunched around zero.

What this means in practice

Xavier initialization was derived assuming roughly linear activations (like tanh near zero) and is the default for layers using tanh or sigmoid; it is built into every framework (nn.init.xavier_uniform_ in PyTorch, GlorotUniform in Keras) so in practice a quant rarely computes it by hand — the risk is using the wrong initializer for the activation function, which reintroduces the exact vanishing/exploding problem this scheme exists to prevent.