The Reparameterization Trick

Training a network by gradient descent needs one thing above all: every operation in the forward pass must have a well-defined slope, so backpropagation can multiply slopes together on the way back and tell every weight which direction to move. A random draw — literally sampling a number from a distribution — has no slope. Asking "if I nudge the mean of this distribution up a hair, how does the sample I just drew change?" doesn't have a sensible answer, because the sample already happened; it's a fixed number now, disconnected from the parameters that produced it. Any network with a sampling step sitting between its parameters and its loss is, as written, un-trainable by gradient descent. The reparameterization trick rewrites the sampling step so the randomness and the parameters are no longer tangled together, restoring a clean slope.

The analogy: a dartboard with a movable target vs a fixed grid plus wind

Imagine training someone to hit a dartboard by moving the board itself toward wherever their darts happen to land — a strange setup, but suppose that's the training signal. If the dart's landing spot is generated by "throw randomly around the board's current position," there's no way to ask "how would the landing spot have changed if I'd moved the board two inches left," because the randomness of the throw is entangled with the board's position in a way that can't be untangled after the fact — the throw already includes both effects mixed together.

Now change the setup: keep a fixed grid on the wall, always throw at the same random offset from a fixed reference dart, generated by a fixed gust of wind that has nothing to do with the board — and only afterward, add the board's current position to that random offset to get the true landing spot. Now moving the board two inches left has an obvious, exact effect: the landing spot moves two inches left too, because the random part (the wind) and the position part (the board) are cleanly separated, added together only at the very end. The reparameterization trick is exactly this separation, applied to sampling from a distribution.

The mechanism: pull the randomness outside

Say a network needs to sample a hidden value $z$ from a normal distribution whose mean $\mu$ and standard deviation $\sigma$ are themselves outputs of the network — computed from the network's weights, so they need gradients flowing through them. The naive way writes

$$z \sim \mathcal{N}(\mu, \sigma^2)$$

which just says "draw $z$ from a normal distribution centered at $\mu$ with spread $\sigma$" — but as an operation in a computational graph, this treats $z$ as coming directly from a black-box sampler, with no differentiable path from $\mu$ or $\sigma$ to $z$. The reparameterization trick rewrites this identical distribution as

$$z = \mu + \sigma \cdot \epsilon, \qquad \epsilon \sim \mathcal{N}(0, 1)$$

Here $\epsilon$ (epsilon) is a random number drawn from a plain, fixed standard normal distribution — the "wind," which has nothing to do with the network's parameters and needs no gradient. And $\mu + \sigma \cdot \epsilon$ is just ordinary arithmetic: multiplication and addition, both perfectly differentiable. The statistical distribution of $z$ is provably identical either way — this is the same normal distribution with the same mean and spread — but the computational graph is completely different. Now $\partial z/\partial \mu = 1$ and $\partial z/\partial \sigma = \epsilon$, both ordinary numbers, so backpropagation can flow straight through to whatever network layers produced $\mu$ and $\sigma$.

The gradient-descent explorer below shows a parameter being nudged step by step to reduce a loss. Reparameterization is what makes that same machinery applicable when one of the "layers" along the way happens to involve randomness — without it, the optimizer would have no slope to follow through that layer at all.

Worked example 1: a gradient with and without the trick

Suppose a tiny network outputs $\mu = 2.0$ and $\sigma=0.5$, and a downstream loss is $L = (z-3)^2$, wanting $z$ close to 3. Sample $\epsilon = 0.4$ (fixed noise, generated once). Then $z = 2.0 + 0.5(0.4) = 2.2$, and $L = (2.2-3)^2 = 0.64$.

Backward pass: $\partial L/\partial z = 2(z-3) = 2(-0.8) = -1.6$. Using the reparameterized form, $\partial z/\partial \mu = 1$ and $\partial z/\partial \sigma = \epsilon = 0.4$, so

$$\frac{\partial L}{\partial \mu} = -1.6 \times 1 = -1.6, \qquad \frac{\partial L}{\partial \sigma} = -1.6 \times 0.4 = -0.64$$

Both are ordinary numbers a network can use to update its weights. Without reparameterizing — if $z$ had been sampled directly as $z \sim \mathcal{N}(\mu,\sigma^2)$ with no rewritten form — there would be no formula connecting $z$ to $\mu$ and $\sigma$ at all, and this step would simply have no gradient to report.

Worked example 2: checking the trick preserves the distribution

Draw five values of $\epsilon$ from a standard normal: $-0.8, 0.2, 1.1, -0.3, 0.5$, with $\mu=10$, $\sigma=2$. The reparameterized samples are $z = 10 + 2\epsilon$: $8.4,\ 10.4,\ 12.2,\ 9.4,\ 11.0$. Their mean is $(8.4+10.4+12.2+9.4+11.0)/5 = 10.28$, close to $\mu=10$ as expected from just five samples, and their spread is visibly close to $2\times$ the spread of the raw $\epsilon$ values. This confirms the rewritten sampling procedure produces the same kind of numbers a direct draw from $\mathcal{N}(10, 4)$ would — the trick changes how the sample is computed, never what distribution it comes from.

What this means in practice

This trick is what makes Variational Autoencoders trainable end-to-end by ordinary backpropagation, rather than needing much noisier gradient-estimation techniques built for non-differentiable randomness. It generalizes wherever a model must sample from a continuous distribution mid-network — for instance, sampling a latent "market regime" in a generative model of price paths and still wanting gradients to flow back into whatever produced that regime's mean and spread.

Practice

1. Given $\mu=5$, $\sigma=1.5$, and $\epsilon=-1.0$, compute $z$ by hand, then find $\partial z/\partial \mu$ and $\partial z/\partial \sigma$.
2. Explain why $\epsilon$ itself never needs a gradient computed for it, even though it directly determines the value of $z$.
3. A colleague proposes reparameterizing a coin-flip (Bernoulli) latent variable the same way as a normal one, writing $z = p + \epsilon$. What goes wrong with this, and what property of the normal distribution's formula fails to carry over?