Wasserstein GAN and Gradient Penalty

Ordinary GAN training suffers a specific, well-diagnosed failure: once the discriminator gets very good at telling real from fake, its output saturates near $0$ or $1$ almost everywhere, and the gradient the generator receives — which flows through the discriminator — shrinks toward zero right when the generator needs guidance most. The generator ends up with almost no signal for how to improve. A Wasserstein GAN (WGAN) sidesteps this by changing what the second network measures, from a probability to a distance, which turns out not to saturate the same way.

The analogy: moving piles of sand

Think of the real data distribution and the generated distribution as two piles of sand of the same total weight but different shapes. The Wasserstein (earth-mover) distance is, informally, the minimum total amount of sand you'd have to shovel, multiplied by how far you'd have to carry each shovelful, to reshape one pile into the other. Unlike a probability of "is this grain real or fake," this distance changes smoothly as the piles get closer or farther apart — it never flattens out to an uninformative constant, so it keeps giving useful directional feedback even when the two piles are already fairly similar.

The critic, and the constraint it needs

WGAN replaces the discriminator with a "critic" $f$ that outputs an unbounded real number rather than a probability, and trains it to approximate the Wasserstein distance via:

$$\max_{f: \lVert f \rVert_L \le 1} \; \mathbb{E}_{x \sim p_{\text{data}}}[f(x)] - \mathbb{E}_{z}[f(G(z))]$$

In words: find the critic that scores real samples as high as possible and generated samples as low as possible, subject to $f$ being 1-Lipschitz — meaning $f$'s output can't change faster than its input does, formally $|f(x_1)-f(x_2)| \le \lVert x_1-x_2\rVert$ for any two points. That Lipschitz constraint is not optional decoration; without it the critic can make the gap between real and fake scores arbitrarily large by scaling itself up, which is meaningless and gives no useful gradient either.

Gradient penalty (WGAN-GP) enforces the constraint softly by penalizing the critic whenever its gradient norm strays from $1$, checked at random points $\hat{x}$ interpolated between real and fake samples:

$$\mathcal{L}_{\text{GP}} = \lambda \, \mathbb{E}_{\hat{x}} \big[ (\lVert \nabla_{\hat{x}} f(\hat{x}) \rVert_2 - 1)^2 \big]$$

In words: sample points between real and fake data, measure how steeply the critic's output changes there, and penalize any deviation from exactly slope $1$ — pushing the critic to stay Lipschitz everywhere it might actually be evaluated, not just at the training points.

Worked example 1: computing the penalty at one point

Suppose at an interpolated point $\hat{x}$ the critic's gradient has norm $\lVert \nabla f(\hat{x})\rVert = 2.5$ and $\lambda = 10$. Penalty contribution: $10 \times (2.5-1)^2 = 10 \times 2.25 = 22.5$ — a large penalty, correctly discouraging the critic from being this steep there. If training brings the gradient norm to $1.1$: penalty $=10\times(0.1)^2=0.1$, nearly nothing, as it should be for a critic that's almost exactly 1-Lipschitz at that point.

Worked example 2: why the Wasserstein estimate itself is informative

Suppose over a batch, $\mathbb{E}[f(x_{\text{real}})] = 4.2$ and $\mathbb{E}[f(G(z))] = 1.6$. The critic's estimate of the Wasserstein distance is $4.2 - 1.6 = 2.6$. Unlike an ordinary GAN's discriminator accuracy (which can sit at 100% while telling you nothing about how different the distributions are), this number is an actual distance estimate: as training proceeds and the generator improves, this gap should shrink toward $0$ smoothly, and — unlike ordinary GAN loss curves — it correlates reasonably well with visual sample quality, which is why WGAN loss curves are usable as a rough training diagnostic in a way vanilla GAN losses are not.

Picture the descent above as the critic's own training: because its objective never saturates the way a discriminator's sigmoid output does, its gradient keeps a useful, non-vanishing slope for far more of training.

What this means in practice

WGAN-GP is close to a default choice whenever GAN training is unstable or mode-collapsing, including in financial applications like QuantGAN-style synthetic return generation, where a vanishing generator gradient would mean the model stops improving on the specific stylized facts (fat tails, volatility clustering) that matter most. The Lipschitz constraint has a real cost, though: it can slightly limit how sharp the critic's decision boundary is allowed to be.