Variational Autoencoders

An ordinary autoencoder compresses each input to one specific point in a latent space, and nothing in its training ever asks those points to be organized sensibly. Two similar inputs might land far apart; the empty gaps between training points might decode into garbage. That's fine if all you want is reconstruction, but it's fatal if you want to generate new examples by picking a fresh point and decoding it — you have no way of knowing which points in that latent space are "valid."

A variational autoencoder (VAE) fixes this by encoding each input not as a single point but as a small cloud of uncertainty around a point — a probability distribution. Picture the difference between pinning a single photograph of a person to a corkboard versus pinning a fuzzy, soft-edged region labeled "roughly here, give or take." When you train the whole board to have every fuzzy region overlap sensibly with its neighbors and blanket the space without gaps, picking any random point on the board — even one that isn't the exact center of anyone's photo — lands you somewhere plausible.

The mechanics, one symbol at a time

The encoder no longer outputs a code $z$ directly; it outputs the parameters of a distribution over $z$ — typically a mean $\mu$ and a spread $\sigma$ for a Gaussian. A sample is then drawn from that distribution and passed to the decoder, and training optimizes two objectives at once:

$$\mathcal{L} = \underbrace{\| x - \hat{x} \|^2}_{\text{reconstruction}} + \beta \cdot \underbrace{D_{\text{KL}}\big(\mathcal{N}(\mu, \sigma^2) \,\|\, \mathcal{N}(0, 1)\big)}_{\text{regularizer}}$$

In words: the first term is the same reconstruction error as an ordinary autoencoder — decode the sampled $z$ and check how close you got back to $x$. The second term, the KL divergence, penalizes the encoder's distribution for straying from a plain, standard bell curve centered at 0 with spread 1; it's a measure of how different two probability distributions are, and pushing it toward zero pulls every input's fuzzy region toward a shared, overlapping, well-organized space rather than letting each input claim its own isolated corner. $\beta$ controls the trade-off: push it up and the latent space becomes tidier and more generative-friendly but reconstructions get blurrier; push it down and you get sharper reconstructions but a messier, less sample-able space.

Sampling from a distribution inside a network that needs to backpropagate is normally impossible — you can't take a gradient through a random draw. VAEs solve this with the reparameterization trick: instead of sampling $z$ directly, sample a fixed random number $\epsilon \sim \mathcal{N}(0,1)$ and compute $z = \mu + \sigma \cdot \epsilon$. The randomness is now isolated in $\epsilon$, which needs no gradient, while $\mu$ and $\sigma$ remain ordinary, differentiable outputs of the encoder.

Worked example 1: encoding, sampling, and decoding one input

An input encodes to $\mu = 1.5$, $\sigma = 0.3$ (a 1-dimensional latent space, for hand computation). Draw a random standard normal sample, say $\epsilon = 0.8$ (a value 0.8 standard deviations above the mean of a standard bell curve — a plausible, not extreme, draw).

$$z = \mu + \sigma \epsilon = 1.5 + 0.3(0.8) = 1.5 + 0.24 = 1.74$$

Decode $z=1.74$ through the decoder — suppose it's the simple linear decoder $\hat{x} = 2.1z - 0.5$: $\hat{x} = 2.1(1.74) - 0.5 = 3.654 - 0.5 = 3.15$. Run the same input through the encoder again and $\epsilon$ will be different each time (it's freshly sampled), giving a slightly different $z$ and thus a slightly different reconstruction — the same input produces a small family of plausible outputs rather than one exact copy, which is the direct, mechanical consequence of encoding to a distribution instead of a point.

Worked example 2: the KL penalty, computed

For a Gaussian $\mathcal{N}(\mu, \sigma^2)$ against the standard normal $\mathcal{N}(0,1)$, the KL divergence has a closed form:

$$D_{\text{KL}} = \frac{1}{2}\left(\mu^2 + \sigma^2 - \ln(\sigma^2) - 1\right)$$

Using $\mu = 1.5$, $\sigma = 0.3$: $D_{\text{KL}} = \frac{1}{2}(2.25 + 0.09 + 2.408 - 1) \approx 1.87$. Compare a better-behaved encoding, $\mu = 0.2$, $\sigma = 0.9$, close to the standard normal: $D_{\text{KL}} = \frac{1}{2}(0.04+0.81+0.211-1) \approx 0.03$, far smaller. The first encoding sits confidently far from the origin with a tight spread — good for accurate reconstruction, bad for the shared latent space the KL term wants; the second costs some reconstruction precision but is close to what the regularizer rewards.

The $-\ln(\sigma^2)$ term inside the KL formula is why a variance collapsing toward zero (an overconfident encoder claiming "this input is exactly this point, no uncertainty") is punished increasingly harshly — the explorer above shows how a log-based penalty blows up as its argument shrinks toward zero, exactly the shape of that term.

What this means in practice

Once trained, a VAE can generate new synthetic examples by skipping the encoder entirely: sample $z$ straight from $\mathcal{N}(0,1)$ and feed it to the decoder. In finance this is used to generate synthetic market scenarios or price paths for stress-testing a strategy against situations that never happened historically but are statistically plausible, and for anomaly detection using the same reconstruction-error logic as a plain autoencoder. The $\beta$ trade-off is the main practical knob: too high and reconstructions get blurry; too low and the latent space degenerates back toward a plain autoencoder's disorganized one, defeating the purpose.