Latent Variable Models and the ELBO

Some datasets look like they were generated by a process with a hidden step nobody observed. A handwritten digit's exact pixels depend on things like the writer's mood, pen pressure, and stroke style — none of which appear in the data, only their downstream effect does. A latent variable model formalizes this by inventing an unobserved variable, call it the hidden cause, and saying the observed data was generated from it. The trouble is fitting such a model requires averaging over every possible value the hidden cause could have taken, which for anything but the simplest models is an integral with no closed form and too many dimensions to brute-force. The ELBO is the trick that sidesteps this integral entirely.

The analogy: guessing a chef's recipe from the plate

Imagine tasting a dish and trying to work out the recipe. The dish (what you observe) could have come from thousands of different recipes (the hidden cause) with slightly different amounts of salt, plated slightly differently. To know how likely your favorite recipe theory is, you'd technically need to consider every possible recipe that could have produced this exact plate and add up their likelihoods — an impossible amount of guessing. Instead, a practical food critic proposes one plausible recipe, tastes critically, and asks "how good is my best guess, and how confident am I in it?" That's not the true, unreachable answer, but it's a workable stand-in — and if the critic keeps refining the guess, their stand-in estimate keeps getting closer to the truth from below, never overshooting it. The ELBO is that stand-in: a quantity you can compute, that is guaranteed never to overstate how well the model explains the data, and that improves as your guess about the hidden cause improves.

The mechanism: a bound instead of the real thing

Call the observed data $x$ (the plate) and the hidden cause $z$ (the recipe). What we actually want is $\log p(x)$, the log-probability the model assigns to the data we observed — high is good, it means the model finds this data unsurprising. Computing it exactly requires

$$p(x) = \int p(x, z)\, dz$$

which sums the joint probability of data-and-hidden-cause over every conceivable value of $z$ — the impossible amount of guessing. Instead, introduce an approximate guesser $q(z \mid x)$ — a distribution over plausible hidden causes given the data, playing the role of the food critic's proposed recipe — and it can be shown through algebra (not "it can be shown that" hand-waving — this comes directly from Jensen's inequality applied to the concavity of the log function) that

$$\log p(x) \;\geq\; \underbrace{\mathbb{E}_{q(z\mid x)}\big[\log p(x \mid z)\big]}_{\text{reconstruction term}} \;-\; \underbrace{D_{\text{KL}}\big(q(z\mid x) \,\|\, p(z)\big)}_{\text{regularization term}}$$

The right-hand side is the ELBO — the Evidence Lower BOund. In words: the true log-probability of the data is at least as large as two computable pieces added together. The first piece asks "if I sample a hidden cause from my guesser $q$ and try to reconstruct the data $x$ from it, how well does that work, on average?" — reward accurate reconstruction. The second piece asks "how far has my guesser $q$ strayed from a simple prior belief $p(z)$ about what hidden causes should look like in general?" — penalize straying too far, which keeps the guesser honest rather than memorizing. Both pieces are computable with real numbers; the original integral never has to be touched.

Maximizing the ELBO instead of the true log-probability is safe precisely because the ELBO is a lower bound: pushing it up can never make it exceed the true value, so improving the bound reliably improves (or at worst plateaus) the real quantity you actually care about.

The convergence-plot explorer below shows a running estimate settling toward a true value as more samples accumulate — the same shape as the ELBO climbing toward $\log p(x)$ across training iterations, tightening step by step rather than jumping there in one move.

Worked example 1: computing the ELBO's two pieces with toy numbers

Suppose for one data point the reconstruction term evaluates to $\mathbb{E}_q[\log p(x\mid z)] = -2.3$ (the model's average log-probability of correctly reconstructing $x$ from a sampled $z$; less negative is better) and the KL penalty evaluates to $D_{\text{KL}}(q \| p) = 0.4$ (the guesser has drifted mildly from the prior). The ELBO is

$$\text{ELBO} = -2.3 - 0.4 = -2.7$$

This says: the true $\log p(x)$ is guaranteed to be at least $-2.7$. If a second, better-trained guesser achieves reconstruction $-1.8$ and KL $0.5$, its ELBO is $-1.8-0.5=-2.3$ — higher (less negative) than before, meaning it is a tighter, more informative lower bound, even though its KL penalty went up. Whichever guesser produces the higher ELBO is preferred, because it is provably closer to the truth from below.

Worked example 2: why a good reconstruction alone isn't enough

Suppose a guesser $q$ achieves excellent reconstruction, $\mathbb{E}_q[\log p(x\mid z)] = -0.1$, by memorizing a very specific $z$ for every single data point — but this $z$ is wildly different from the prior's expectations, giving $D_{\text{KL}}(q\|p) = 15.0$. The ELBO is $-0.1 - 15.0 = -15.1$ — far worse than a guesser with modest reconstruction $-2.0$ and a small KL of $0.3$, giving ELBO $-2.3$. The KL term exists precisely to punish this kind of memorization: a hidden-cause distribution that has collapsed onto a narrow answer for each data point stops behaving like a distribution at all and stops being able to generate anything new by sampling from the prior.

What this means in practice

The ELBO is the training objective behind variational autoencoders and much of modern generative modeling: it's how a model learns to compress data into a compact hidden representation while still being able to generate plausible new samples from that representation. In finance, latent variable models of this kind underlie synthetic scenario generation — learning a compact hidden "market regime" variable that explains observed price paths, then sampling new regimes to stress-test a portfolio against scenarios that never literally occurred.

Practice

1. Two guessers achieve ELBOs of $-3.1$ and $-2.9$. Which is closer to the true $\log p(x)$, and can you tell from these numbers alone how close?
2. Explain in one sentence why the KL term is written as a penalty (subtracted) rather than added to the reconstruction term.
3. If the KL term were removed entirely from training, what would you expect to happen to the hidden-cause distribution $q(z\mid x)$ learned for each data point, and why would that break the model's ability to generate new samples?