Latent Diffusion Models

A standard diffusion model runs dozens to hundreds of denoising steps, and each step is a full forward pass of a large network over the entire data — every pixel of a high-resolution image, say. Most of that computation is spent on fine, perceptually minor detail that doesn't affect the overall content of the image much. Latent diffusion's insight: do the expensive, many-step diffusion process in a much smaller compressed space instead, and only pay the cost of full resolution once, at the very end, when a separately-trained decoder expands the result back out.

The analogy: editing a thumbnail, then upscaling once

Imagine editing a large photo by first shrinking it to a thumbnail, making every compositional change — cropping, color balance, retouching — on the tiny thumbnail where each edit is nearly instant, and only enlarging back to full resolution once you're satisfied, using a separate high-quality upscaler that fills back in fine detail. You do all the expensive, iterative decision-making cheaply, and pay the cost of full resolution exactly once, at the end, rather than on every single edit along the way.

The architecture: compress, diffuse, decode

Latent diffusion uses three pieces, trained mostly separately: an encoder $\mathcal{E}$ compressing data $x$ into a smaller latent $z = \mathcal{E}(x)$; a diffusion model that runs its entire forward-noising and reverse-denoising process on $z$, not $x$; and a decoder $\mathcal{D}$ mapping the final denoised latent back to data space, $\hat{x} = \mathcal{D}(z_0)$. The diffusion loss is the same denoising objective as ordinary diffusion, just evaluated in latent space:

$$\mathcal{L} = \mathbb{E}_{z_0, t, \epsilon}\Big[ \big\lVert \epsilon - \epsilon_\theta(z_t, t) \big\rVert^2 \Big], \qquad z = \mathcal{E}(x)$$

In words: at a random noise level $t$, add noise $\epsilon$ to the latent $z_0$ to get $z_t$, ask the network to predict that noise back out, and penalize the error — identical in form to ordinary diffusion, but every quantity lives in the small latent space instead of pixel (or price-path) space.

Worked example 1: the compute saving, in numbers

Suppose a 512×512×3 image has $512 \times 512 \times 3 \approx 786{,}000$ values, and the autoencoder compresses by a factor of 8 in each spatial dimension down to a 64×64×4 latent, $64\times64\times4 \approx 16{,}400$ values — about 48x fewer values per diffusion step. If a single denoising network pass costs roughly proportional to input size, and generation requires 50 such passes, pixel-space diffusion pays that full cost 50 times over $786{,}000$-sized inputs, while latent diffusion pays it 50 times over $16{,}400$-sized inputs plus one single encode and one single decode pass at full resolution — the 50 expensive iterative steps get almost 50x cheaper, at the cost of two extra full-resolution passes total.

Worked example 2: why the latent has to be a good compression first

Suppose the autoencoder is trained poorly and its latent space loses fine structure the decoder can't recover — reconstructing a clean image with no diffusion involved already yields visible blur or missing detail, say a reconstruction error equivalent to 15% pixel-wise deviation. No matter how well the diffusion model performs within that latent space (even a perfect denoiser reaching the exact right $z_0$), the final output is capped by that same 15% reconstruction gap, because the decoder simply cannot produce detail the encoder discarded. This is why latent diffusion models train the autoencoder first, to a high reconstruction standard, before ever training the diffusion model on top of it — diffusion quality can never exceed the autoencoder's own reconstruction ceiling.

Think of the compression factor here the way you'd think of a compounding rate applied in reverse — instead of one input growing over many steps, latent diffusion shrinks the space each step operates in once, up front, so every one of the many repeated steps that follow is cheap by the same multiplicative factor.

What this means in practice

Latent diffusion is why modern high-resolution image and audio diffusion models are computationally feasible at all; the same architecture generalizes to any high-dimensional financial data — a full limit order book snapshot or a high-frequency multi-asset panel — where compressing to a smaller learned representation before running iterative generation makes an otherwise-prohibitive compute budget practical.