Variational Inference

Gibbs sampling and other MCMC methods get you an exact posterior, eventually — but "eventually" can mean hours or days for a model with millions of parameters, like a large ranking or recommendation model updated every trading day. Sometimes you need an answer in seconds, and you're willing to trade some accuracy for it. Variational inference is that trade: instead of drawing samples that gradually paint a picture of the true posterior, it picks a family of simple, easy-to-work-with distributions — independent normals, say — and directly optimizes to find the member of that family closest to the true posterior. No sampling, no waiting for a chain to converge — just an optimization problem, which is usually much faster to solve.

An analogy: fitting a rug to an oddly-shaped room

Imagine the true posterior is the exact, irregular floor plan of a room — full of alcoves and curves that no simple shape captures perfectly. MCMC is like carefully placing thousands of tiles across the whole floor, one by one, until their arrangement traces the true shape as closely as you like — accurate, but slow. Variational inference instead asks: "what's the best-fitting simple rectangular rug I can lay down, chosen from a catalog of rectangles, that covers as much of this room as well as possible?" You'll never get the alcoves exactly right, and the rug you pick is guaranteed to be simpler than the room — but you get an answer to "roughly what does this room look like" in the time it takes to measure four corners, not the time it takes to lay a thousand tiles.

The method, one symbol at a time

Let $\theta$ be the parameters and $p(\theta \mid \text{data})$ the true (intractable) posterior. Choose a family of simpler distributions $q(\theta; \phi)$ ("q of theta, indexed by phi") — for instance, a normal distribution, where $\phi$ ("phi") stacks together its mean and variance. Variational inference finds the $\phi$ that makes $q$ closest to $p$, where "closest" is measured by the Kullback-Leibler divergence:

$$\phi^{*} = \arg\min_{\phi}\ \text{KL}\big(q(\theta;\phi)\ \|\ p(\theta \mid \text{data})\big) .$$

In words: search over every candidate distribution in the simple family, and pick the one, $\phi^*$, that has the smallest KL divergence — a way of scoring how much probability mass one distribution puts where the other doesn't — to the true posterior. Because the true posterior itself is unknown (that's the whole problem), you can't compute the KL divergence directly. The trick is that minimizing it is mathematically equivalent to maximizing a different, computable quantity called the evidence lower bound (ELBO):

$$\text{ELBO}(\phi) = E_{q}\big[\ln p(\text{data}, \theta)\big] - E_{q}\big[\ln q(\theta;\phi)\big] .$$

In words: the ELBO rewards $q$ for putting probability mass where the data and prior jointly say it's likely (the first term), and separately rewards $q$ for staying spread out and uncertain rather than collapsing to a single point (the second term, an entropy bonus) — maximizing this quantity over $\phi$ is a standard optimization problem, solvable with the same gradient-based tools used to train any other statistical model, and it provably pushes $q$ toward $p$ without ever needing $p$ itself.

A common simplifying choice, mean-field variational inference, assumes $q$ factorizes into independent pieces, one per parameter: $q(\theta_1, \dots, \theta_k) = q_1(\theta_1)\cdots q_k(\theta_k)$. In words: assume the approximating distribution treats every parameter as independent of every other, which makes the optimization tractable but throws away any true correlation between parameters — a specific, known cost of the approximation, not a hidden one.

Worked example 1: approximating a skewed posterior with a Gaussian

Suppose the true posterior for a win-rate parameter (from few observed wins, hence right-skewed, as in the credible-intervals page) is $\text{Beta}(3, 39)$, with true mean $3/42 = 0.0714$ and true variance $\dfrac{3 \times 39}{42^2 \times 43} = \dfrac{117}{75852} = 0.00154$ (standard deviation $\approx 0.0393$). A mean-field variational fit restricted to Gaussians would match these first two moments as closely as the optimization allows, landing near $q^* \approx N(0.071,\ 0.00154)$. Compare the two: the true Beta posterior gives essentially zero probability to negative win rates (it's naturally bounded at 0), but the Gaussian approximation, being symmetric and unbounded, assigns roughly $\Pr(\theta<0) = \Phi(-0.071/0.0393) = \Phi(-1.81) \approx 3.5\%$ probability to an impossible negative win rate. That 3.5% is not a computational error — it is the family-choice error worked example flagged in the analogy: the rug is a rectangle, the room has a curve, and no amount of optimizing the rectangle's position fixes that mismatch.

Worked example 2: ELBO improving over iterations

A simplified gradient-based fit tracks the ELBO across iterations as it climbs toward its maximum: iteration 1, $\text{ELBO} = -142.3$; iteration 10, $\text{ELBO} = -98.7$; iteration 50, $\text{ELBO} = -81.2$; iteration 200, $\text{ELBO} = -80.9$; iteration 500, $\text{ELBO} = -80.9$ (unchanged to one decimal). The ELBO rising and then flattening out is the direct, computable signal that $q$ has converged to the best approximation available within its family — you never see the true posterior or the KL divergence directly, but watching the ELBO plateau is the standard, practical stopping rule, exactly analogous to watching a training loss curve flatten when fitting any other model by gradient descent.

The bell curve above is the shape the Gaussian variational family is restricted to — no matter how well the optimization runs, the fitted approximation can only ever look like this shape, symmetric and unbounded, never the true skewed, bounded Beta posterior underneath it.

What this means in practice

• Large-scale Bayesian models in production — recommendation systems, topic models, large hierarchical models refit daily — typically use variational inference or its stochastic variant precisely because MCMC's per-update cost is too high to run on a daily cadence at that scale.
• Variational posteriors systematically understate uncertainty. The KL divergence used above penalizes $q$ heavily for putting mass where $p$ has none, which pushes the fitted $q$ to be narrower than the truth, especially under mean-field independence assumptions — a known, directional bias, not a random error, so credible intervals built from a variational posterior tend to be too tight.
• Choose the family to match what you actually need. If you only need the posterior mean, the family-mismatch cost in worked example 1 barely matters; if you need an accurate tail probability (a VaR-style question), the mismatch can be the whole ballgame.
• Compare to MCMC on a small version of the problem first. Before trusting a variational fit at production scale, validate it against exact Gibbs or Metropolis-Hastings samples on a scaled-down version where both are feasible to run.