RMSNorm and Scale-Only Normalization

Layer normalization takes each example's vector of activations and standardizes it: subtract the mean (recenter to 0) and divide by the standard deviation (rescale to unit spread), then let two learned parameters stretch and shift it back to whatever scale training actually needs. That works well, but computing a mean, then a variance around that mean, then dividing, is three passes of arithmetic over every activation, on every layer, of every token — a real cost at the scale of a modern language model. RMSNorm asks: how much of that is actually necessary?

The analogy: adjusting volume without re-centering pitch

Imagine adjusting the loudness of a recording. There are two separate things you could do: shift every sample up or down by a constant (changing where "silence" sits — recentering), and scale every sample by a factor (changing how loud the loudest parts are — rescaling). If a recording is already reasonably centered around zero and the only real problem is that its volume drifts unpredictably, you don't need the recentering step at all — a single volume knob does the job at half the effort. RMSNorm is betting that most of layer normalization's benefit for deep transformers comes from controlling scale, and that recentering was buying comparatively little.

The formulas, side by side

LayerNorm, for an activation vector $x = (x_1, \dots, x_n)$ with mean $\mu$ and standard deviation $\sigma$:

$$\text{LN}(x_i) = \gamma \cdot \frac{x_i - \mu}{\sigma} + \beta$$

In words: subtract the mean, divide by the standard deviation, then apply a learned scale $\gamma$ and shift $\beta$.

RMSNorm skips the mean subtraction and divides only by the root-mean-square:

$$\text{RMS}(x) = \sqrt{\frac{1}{n}\sum_{i=1}^n x_i^2}, \qquad \text{RMSNorm}(x_i) = \gamma \cdot \frac{x_i}{\text{RMS}(x)}$$

In words: measure the typical magnitude of the vector by averaging its squared entries and taking the square root (never subtracting a mean first), divide every entry by that number, and apply a single learned scale $\gamma$ — there is no learned shift $\beta$ and no recentering step at all.

Worked example 1: RMSNorm by hand

Take $x = (3, -1, 4, 2)$. Sum of squares: $9 + 1 + 16 + 4 = 30$. Mean of squares: $30/4 = 7.5$. RMS $= \sqrt{7.5} = 2.739$. Divide each entry: $(3/2.739,\ -1/2.739,\ 4/2.739,\ 2/2.739) = (1.095,\ -0.365,\ 1.461,\ 0.730)$. With $\gamma=1$, that's the RMSNorm output — no mean was ever computed or subtracted.

Worked example 2: comparing to LayerNorm on the same vector

Same $x = (3, -1, 4, 2)$. Mean $\mu = 2$. Variance $= \frac{1}{4}[(1)^2+(-3)^2+(2)^2+(0)^2] = 3.5$, so $\sigma = 1.871$. LayerNorm output (with $\gamma=1,\beta=0$): $(0.534,\ -1.604,\ 1.069,\ 0)$. The outputs differ meaningfully from RMSNorm's $(1.095, -0.365, 1.461, 0.730)$ — LayerNorm shifted everything relative to the mean of 2 first, RMSNorm never did. Both still succeed at the main goal: neither output has entries that have grown or shrunk to an extreme scale, which is what keeps a deep stack of these layers numerically stable.

What this means in practice

RMSNorm is the default normalization in most contemporary transformer architectures because it removes roughly a third of the arithmetic (no mean, no $\beta$ parameter) while empirically matching LayerNorm's training stability on large models; it's the reason a quant reading a modern LLM's architecture diagram sees "RMSNorm" rather than "LayerNorm" at nearly every block. The saved computation is genuinely material at the scale of billions of parameters and trillions of training tokens, even though the difference in behavior on any single small vector, as above, looks modest.