Batch Normalization

Train a deep network and every layer is chasing a moving target. Layer 5's inputs are layer 4's outputs, and layer 4's weights are changing on every step of training. So the distribution of numbers layer 5 has to cope with — their scale, their spread — keeps shifting under it, even though layer 5's own weights haven't changed. Each layer is like a batter trying to hit a pitch whose speed and angle change every swing, not because the batter is doing anything wrong, but because the pitcher keeps adjusting. Training becomes slow and touchy: push the learning rate up to speed things along and the whole stack can blow up.

Batch normalization fixes the pitch, not the batter. Before each layer's output is passed on, it is rescaled so it always arrives with roughly zero mean and unit spread, batch by batch, during training.

The mechanics, one symbol at a time

Take a mini-batch of $m$ examples and look at one number flowing through the network — say, the pre-activation score $z$ of a single hidden unit, computed for each of the $m$ examples in the batch. Batch norm standardizes it:

$$\hat{z}_i = \frac{z_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}, \qquad y_i = \gamma \hat{z}_i + \beta$$

In words: $\mu_B$ is the average of $z$ across the batch, $\sigma_B^2$ is the batch's variance, and $\epsilon$ is a tiny constant added purely so you never divide by zero. The first equation says "subtract the batch's average and divide by the batch's spread" — exactly the same standardizing you'd do to a column of data before regressing on it, just done fresh at every layer, every step. The second equation says the network is then allowed to undo that standardizing if it wants to, by learning its own scale $\gamma$ and shift $\beta$ per unit. That matters: forcing every unit's output to have exactly unit variance forever would remove information the network might need, so batch norm gives the layer a dial to stretch or shift back and hands the network the freedom to use it or not.

Worked example 1: standardizing one unit across a batch

A hidden unit produces these pre-activation scores for a batch of four training examples: $z = (2, 4, 4, 6)$.

Mean: $\mu_B = (2+4+4+6)/4 = 4$.

Variance: $\sigma_B^2 = \frac{(2-4)^2+(4-4)^2+(4-4)^2+(6-4)^2}{4} = \frac{4+0+0+4}{4} = 2$.

Standard deviation: $\sqrt{2 + \epsilon} \approx 1.414$ (take $\epsilon \approx 0$ for cleanliness).

Standardized values: $\hat{z} = (\frac{2-4}{1.414}, \frac{4-4}{1.414}, \frac{4-4}{1.414}, \frac{6-4}{1.414}) = (-1.41, 0, 0, 1.41)$.

If the layer has learned $\gamma = 2$, $\beta = 0.5$: outputs are $y = 2\hat{z} + 0.5 = (-2.32, 0.5, 0.5, 3.32)$. Same relative shape, rescaled to whatever range the network has decided is useful downstream.

Worked example 2: why this helps training

Suppose two units feed the next layer, one with raw outputs typically around $1000$ and one typically around $0.01$ (unnormalized, because upstream weights happened to grow that way). A single learning rate has to move both. Make it big enough to nudge the second unit meaningfully and it will send the first unit into wild swings; make it small enough to be safe for the first and the second barely learns. After batch norm, both units arrive at the next layer standardized to roughly the same scale — mean near 0, spread near 1 — so one learning rate works reasonably well for both. This is the direct mechanism, not a side effect: same-scale inputs are why the network tolerates larger, more uniform learning rates and trains faster.

Use the explorer above to see how sensitive a sigmoid-style unit is to the scale of its input: a steep, badly-scaled input saturates the unit near 0 or 1, where the gradient is nearly flat and learning stalls. Batch norm's whole job is to keep inputs in the responsive middle of that curve.

What this means in practice

At test time there is no "batch" — you may be scoring one example at a time. So batch norm keeps a running average of $\mu_B$ and $\sigma_B^2$ collected during training and uses those fixed numbers at inference, rather than recomputing them per prediction. This is also why batch norm interacts badly with very small batches or with sequences of correlated batches (a common trap in time series work): a batch of size 2 gives a wildly noisy estimate of the mean and variance, which injects noise into training the model. For sequence models and small-batch financial time series, alternatives like layer normalization (normalizing across features for one example, not across examples in a batch) are usually preferred for exactly this reason.