The Internal Covariate Shift Debate

Batch normalization was introduced in 2015 with a specific explanation for why it works: as a network trains, the distribution of each layer's inputs keeps shifting because the parameters of every earlier layer are changing too, forcing each layer to constantly re-adapt to a moving target — a problem the original paper called "internal covariate shift." Normalizing each layer's inputs to fixed mean and variance was proposed as the fix, stabilizing that moving target and letting training proceed faster.

A well-known 2018 paper directly tested this explanation and found it didn't hold up: the researchers measured internal covariate shift directly (with and without batch normalization) and found batch-normalized networks didn't actually have noticeably less of it, yet still trained faster and more stably. Digging further, they found the real mechanism appeared to be different: batch normalization measurably smooths the loss landscape, making the gradient more predictable and allowing larger, more stable learning rates, whether or not it actually reduces the shifting-distributions problem it was originally designed to fix.

The debate is a useful cautionary tale in deep learning research more broadly: a technique working well in practice doesn't confirm the mechanism originally proposed to explain it, and rigorous ablation studies sometimes overturn intuitive explanations that had been widely repeated for years. Batch normalization remains a standard, effective tool — the disagreement is only about why it works, not whether it works.