The Evidence Lower Bound

Suppose you fit a model with a hidden variable — a regime nobody labelled, a factor nobody observed directly. Bayes' rule says the right belief about that hidden variable, given the data, is the posterior $p(z \mid x)$. Writing it down exactly requires dividing by $p(x)$, the "evidence": the total probability of the data averaged over every possible value of the hidden variable. For anything beyond a toy model, that average is an integral over a space too large to sum, and it has no closed form. You know the rule but cannot evaluate it.

The analogy: appraising a house you cannot fully inspect

You want to know a house's true value. You cannot get inside every room, so an exact appraisal is out of reach. What you can do is walk the perimeter, check the neighbourhood, and build a conservative valuation from what's visible — a number you can defend as a floor: "it's worth at least this much." A better inspector, with a richer checklist, pushes that floor closer to the true value without ever proving it exact.

The evidence lower bound, or ELBO, is that floor. You cannot compute $\log p(x)$, the log-evidence, directly. You can compute a related quantity that is always less than or equal to it, and the closer your approximating "checklist" gets to reality, the tighter that floor becomes.

Building the bound

Introduce an approximating distribution $q(z)$ — a guess at the posterior, chosen from a family simple enough to compute with, such as a Gaussian with parameters you control. A short algebraic identity (splitting $\log p(x)$ using $p(x,z) = p(z \mid x) p(x)$) gives:

$$\log p(x) = \mathbb{E}_{q(z)}\!\left[\log \frac{p(x,z)}{q(z)}\right] + \mathrm{KL}\big(q(z) \,\|\, p(z \mid x)\big)$$

In words: the true log-evidence splits into two pieces. The first piece — the ELBO — is an average, under your guess $q$, of the log joint probability of data and hidden variable minus the log of your guess itself. The second piece is the KL Divergence between your guess and the true posterior, which measures how wrong your guess is. Crucially, KL divergence is never negative, so the ELBO can never exceed $\log p(x)$: it is a genuine lower bound, and the gap between them is exactly your guess's error.

$$\mathrm{ELBO}(q) = \mathbb{E}_{q(z)}[\log p(x,z)] - \mathbb{E}_{q(z)}[\log q(z)]$$

In words: reward $q$ for placing mass where the data and hidden variable are jointly plausible (first term), and reward it separately for not collapsing to a single point — the second term is $q$'s own entropy, and a confident, low-entropy guess that's wrong gets penalised harder than a spread-out one. Since $\log p(x)$ is a fixed number that does not depend on $q$, maximising the ELBO over $q$ is mathematically identical to minimising the KL gap. You never touch the intractable quantity directly; you climb a bound instead, and the true posterior is what you land on when the family containing $q$ is rich enough to reach it exactly.

Worked example 1: a two-outcome hidden variable, by hand

A hidden variable $z$ takes value 1 (a "risk-on" regime) or 0 ("risk-off") with true posterior $p(z{=}1 \mid x) = 0.8$ after seeing today's data — but suppose, for illustration, you cannot compute this directly and only know the unnormalised joint values $p(x, z{=}1) = 0.32$ and $p(x, z{=}0) = 0.08$, so the true evidence is $p(x) = 0.32 + 0.08 = 0.40$, giving $\log p(x) = \log(0.40) = -0.916$.

Guess $q(z{=}1) = 0.6$, $q(z{=}0) = 0.4$ — a deliberately imperfect approximation. Compute the ELBO term by term:

$$\mathbb{E}_q[\log p(x,z)] = 0.6 \log(0.32) + 0.4 \log(0.08) = 0.6(-1.139) + 0.4(-2.526) = -0.683 - 1.010 = -1.693$$ $$\mathbb{E}_q[\log q(z)] = 0.6\log(0.6) + 0.4\log(0.4) = 0.6(-0.511) + 0.4(-0.916) = -0.307 - 0.367 = -0.673$$ $$\mathrm{ELBO} = -1.693 - (-0.673) = -1.020$$

Check: $-1.020 \le -0.916$, so the bound holds, with a gap of $0.104$ — that gap is $\mathrm{KL}(q \| p)$, and it is nonzero because $q(z{=}1)=0.6$ doesn't match the true $p(z{=}1 \mid x) = 0.32/0.40 = 0.8$.

Worked example 2: tightening the guess

Now try $q(z{=}1) = 0.8$, $q(z{=}0) = 0.2$ — matching the true posterior exactly.

$$\mathbb{E}_q[\log p(x,z)] = 0.8(-1.139) + 0.2(-2.526) = -0.911 - 0.505 = -1.416$$ $$\mathbb{E}_q[\log q(z)] = 0.8\log(0.8) + 0.2\log(0.2) = 0.8(-0.223) + 0.2(-1.609) = -0.178 - 0.322 = -0.500$$ $$\mathrm{ELBO} = -1.416 - (-0.500) = -0.916$$

That equals $\log p(x)$ exactly. When $q$ matches the true posterior, the KL gap vanishes and the floor touches the ceiling — this is the whole mechanism variational inference exploits: search over $q$ for whatever maximises the ELBO, and you are, without ever computing $p(x)$, searching for the best available approximation to $p(z \mid x)$.

What this means in practice

Nobody in production computes an ELBO by hand over two outcomes — the real payoff is when $z$ is continuous and high-dimensional, such as the latent factors in a variational autoencoder used for anomaly detection on order-book snapshots, or the latent regime in a state-space model of returns. There, $p(x)$ truly has no closed form, and optimising the ELBO with gradient ascent (via the reparameterisation trick, so gradients can flow through the sampling step) is the only practical route to an approximate posterior. It also underlies model selection: the ELBO is sometimes used as a cheap stand-in for the evidence when comparing model families, since a model whose best achievable ELBO is higher tends to explain the data better — though this comparison inherits any bias from how tight each model's bound actually is.

Watch how the tracked quantity climbs and settles as more updates accumulate — the ELBO under gradient ascent behaves the same way, rising each step and flattening out once $q$ stops improving, never above the true log-evidence.