Spectral Normalization for GAN Stability

A GAN's discriminator is trained to tell real data from the generator's fakes, and the generator improves by using the discriminator's gradient as a training signal. If the discriminator gets too sharp — its output swings wildly for tiny changes in input — that gradient becomes useless or explosive, and training collapses. The technical way to say "not too sharp" is that the discriminator's Lipschitz constant should be bounded: its output shouldn't change faster than some fixed rate relative to how much its input changed.

Spectral normalization enforces this directly on every weight matrix in the discriminator, layer by layer. Each weight matrix $W$ is divided by its spectral norm — its largest singular value, $\sigma(W)$ — so the rescaled matrix $W / \sigma(W)$ has spectral norm exactly $1$, meaning it can stretch any input vector by at most a factor of $1$. Since a network's overall Lipschitz constant is bounded by the product of its layers' individual Lipschitz constants, normalizing every layer this way caps the whole discriminator's sensitivity in one mechanical step, with no extra loss terms or hyperparameters to tune.

Computing the exact largest singular value every training step would be expensive, so in practice it's approximated cheaply with one or two iterations of the power method, updated alongside the weights each step — accurate enough to keep training stable at negligible extra cost. This became one of the standard, close-to-default stabilization tricks for GAN discriminators after 2018, often paired with architectural choices like those in DCGAN.