Group Normalization

Batch normalization stabilizes neural network training by normalizing each channel's activations using the mean and variance computed across a whole batch of examples. That works well with large batches, but falls apart with small ones — a batch of size 2 or 4 gives a noisy, unreliable estimate of the mean and variance, which is common in tasks like high-resolution image processing where memory limits force small batches. Group normalization sidesteps the problem by never looking across examples at all: for each individual example, it splits the channels into a fixed number of groups and normalizes within each group, using only that one example's own activations.

Because the statistics are computed per example rather than per batch, group normalization gives the same result whether you run one example at a time or a thousand — its behavior doesn't depend on batch size, which makes it attractive for architectures like image segmentation or detection networks where large batches aren't practical.

Worked example. A layer has 32 channels. With group size set to 8, group normalization creates 4 groups of 8 channels each, and for a single training example computes one mean and variance per group (4 pairs total), normalizing every channel using its own group's statistics — no other examples in the batch are ever consulted.