Pre-Norm vs Post-Norm Transformer Blocks

Every transformer block wraps its attention and feedforward sub-layers in a residual connection — the sub-layer's output is added to its input, not used to replace it — plus a layer normalization step that rescales activations to keep training stable. The original transformer placed normalization after the addition. Later work found that simply moving it to before the sub-layer instead changes how gradients flow through a deep stack of these blocks so much that it is the difference between a 100-layer transformer training smoothly and one that barely trains at all without extremely careful, slow learning-rate warmup.

The analogy: renovating a house room by room versus renovating the blueprint

Post-norm is like renovating a house room by room and re-leveling the entire foundation after every room finishes, before starting the next — each renovation compounds on a foundation that was just entirely reset, so an early error gets absorbed into that reset, hard to trace later. Pre-norm is like leveling only the specific room you're about to work on, then adding its result straight onto the house without re-leveling the whole foundation — the original foundation (the accumulating residual stream) stays intact and traceable throughout.

The mechanics: where the normalization sits relative to the addition

Post-norm (the original transformer): normalize after adding the sub-layer's output to the residual stream.

$$x_{\ell+1} = \text{LayerNorm}\big(x_\ell + \text{Sublayer}(x_\ell)\big)$$

Pre-norm: normalize the input before feeding it to the sub-layer, and add the sub-layer's output straight onto the unnormalized residual stream.

$$x_{\ell+1} = x_\ell + \text{Sublayer}\big(\text{LayerNorm}(x_\ell)\big)$$

In words: in post-norm, every block's output gets rescaled, so the running total (residual stream) is repeatedly reset in scale. In pre-norm, the running total is never rescaled — only a temporary, normalized copy feeds the sub-layer, while the actual sum keeps growing untouched. That matters most for the gradient: in pre-norm, the gradient w.r.t. $x_\ell$ has a clean, direct identity path back to the first layer, with sub-layer gradients only ever added on top, never multiplied in. In post-norm, every layer's normalization sits directly on that backward path, repeatedly rescaling gradients — in a very deep stack, this can shrink gradients toward the earliest layers to nearly nothing.

Worked example 1: tracing a gradient through 3 blocks by hand

Suppose a 3-block pre-norm stack, gradient of loss w.r.t. final output is $1.0$. Because pre-norm's residual path is a pure identity, the gradient reaching $x_0$ is at least $1.0$ from the identity path alone, plus whatever flows through the sub-layer branches. In a post-norm stack, each of the 3 layer-norms multiplies the gradient by some factor typically less than 1 — if each norm's effective backward factor is 0.6, the gradient reaching $x_0$ from that path alone shrinks to roughly $0.6^3 \approx 0.22$, less than a quarter its starting size.

Worked example 2: why this compounds badly at real depth

Extend the same 0.6-per-layer shrinkage in a post-norm stack to 24 layers (a modest real transformer): $0.6^{24} \approx 0.0000022$ — the gradient reaching the earliest layers is effectively zero, and those layers barely update without intervention. This is precisely why the original post-norm transformer needed a careful learning-rate warmup schedule to avoid diverging before normalization statistics stabilized. A pre-norm stack at the same 24 layers keeps a clean identity gradient path regardless of depth, which is why pre-norm transformers train reliably with simpler schedules and scale to much greater depth.

Picture the identity path in pre-norm as a direct, unobstructed line from output back to input — like taking clean, well-sized steps toward a minimum — versus post-norm's repeatedly-rescaled path, more like steps that shrink at every layer until they barely move the earliest parameters at all.

What this means in practice

Nearly every modern large transformer (GPT-2 onward, and most large language models since) uses pre-norm specifically because it trains more reliably at depth and tolerates simpler optimization schedules, even though the original 2017 transformer paper used post-norm. The tradeoff is not free — pre-norm's unnormalized, ever-growing residual stream can itself grow to very large magnitudes at great depth, which some newer normalization variants (like a final normalization layer at the very end of the stack) are specifically designed to control.

Practice

1. Write the pre-norm and post-norm update equations side by side and identify exactly which term is normalized in each.
2. If each post-norm layer shrinks backward gradient by a factor of 0.8, what is the approximate gradient scale reaching the input of a 12-layer stack?
3. Explain in one sentence why pre-norm's residual stream can grow very large in magnitude at great depth.