Weight Normalization

Training a deep network is partly a fight against badly scaled gradients: a weight vector's direction (which input features it emphasizes) and its magnitude (how strongly it fires overall) get tangled together in a single gradient step, so improving one often disturbs the other and slows convergence. Weight normalization untangles them by rewriting each weight vector $\mathbf{w}$ as $\mathbf{w} = g \cdot \mathbf{v}/\|\mathbf{v}\|$, where $\mathbf{v}$ is a direction vector and $g$ is a single learned scalar for magnitude. Gradient descent then updates direction and scale as separate, decoupled quantities instead of one entangled vector, which in practice makes the loss surface easier to descend and reduces sensitivity to the initial scale of the weights.

Think of steering a car: instead of one control that mixes "how far to turn" and "how hard to accelerate," you get two clean, independent dials. Adjusting your heading no longer accidentally changes your speed, so each dial can be tuned to its own natural pace.

Unlike batch normalization, weight normalization doesn't depend on statistics computed across a mini-batch — it's a pure reparameterization of the weights themselves, computed independently for each example. That makes it well suited to settings where batch statistics are noisy or unavailable, such as reinforcement learning with small batches, recurrent networks, or generative models, and it adds negligible computational overhead since it only introduces one extra scalar parameter per weight vector.