Layer Normalization

Deep networks train poorly when the numbers flowing through each layer drift to very large or very small scales, because gradients then explode or vanish. Layer normalization fixes this by rescaling the activations within a single training example, one layer at a time, so each layer always sees inputs on a similar, predictable scale regardless of what came before.

Concretely, for one example's activations $x = (x_1, \dots, x_d)$ in a layer, it computes the mean $\mu$ and standard deviation $\sigma$ across those $d$ values, then normalizes:

$$\hat{x}_i = \frac{x_i - \mu}{\sigma + \epsilon}, \qquad y_i = \gamma \hat{x}_i + \beta,$$

where $\epsilon$ is a tiny constant avoiding division by zero, and $\gamma, \beta$ are learned scale and shift parameters that let the network undo the normalization if that turns out to help. In plain English: subtract the average and divide by the spread, then let the model relearn whatever scale it actually wants.

The key difference from batch normalization is what gets averaged over: batch norm computes $\mu,\sigma$ across all examples in a mini-batch (so it depends on batch size and breaks with a batch of one), while layer norm computes them across the features of one single example (so it works identically whether you process one sequence at a time or a thousand). This is why layer norm is the default in transformers and sequence models, where batch sizes vary and sequences are processed one token-position at a time.

Worked example. Given one example's pre-activation values $x = (2, 4, 6)$: $\mu = 4$, and the standard deviation is $\sigma = \sqrt{\frac{(2-4)^2+(4-4)^2+(6-4)^2}{3}} = \sqrt{8/3} \approx 1.63$. So $\hat{x} \approx (-1.22, 0, 1.22)$ — same shape, now centered at zero with unit-ish spread, before $\gamma,\beta$ rescale it.