Choosing Between Variational Inference and MCMC

Bayesian inference asks for the full posterior distribution over a model's parameters given data, but for almost any realistic model that posterior has no clean formula — it has to be approximated. There are two fundamentally different ways to do that, and picking wrong wastes either a research afternoon or a week of compute.

The analogy: a fast sketch versus a surveyed map

Imagine mapping an unfamiliar coastline. One approach is to fly over it and quickly sketch a smooth curve that looks about right — fast, cheap, but it will straighten out real inlets and miss anything the sketching shape (say, a simple oval) cannot represent. The other approach is to walk the entire coast on foot, logging your GPS position every few steps — slow and expensive, but given enough steps, the recorded path is the coastline, no matter how jagged. Variational inference (VI) is the aerial sketch: it picks a simple, tractable family of shapes (like independent Gaussians) and finds the member of that family closest to the true posterior. MCMC (Markov chain Monte Carlo) is the walk: it draws a long sequence of correlated samples that, given enough of them, come from the true posterior exactly, whatever shape it happens to have.

What each one actually optimizes

VI turns inference into an optimization problem: pick a distribution $q$ from a simple family and maximize the ELBO, which is equivalent to minimizing the KL divergence between $q$ and the true posterior $p$. Because $q$ is restricted to a simple shape, VI is fast (often converging in seconds to minutes) but it is biased — if the true posterior is not shaped like anything in $q$'s family, VI cannot represent it, however long it runs.

MCMC turns inference into a sampling problem: build a Markov chain whose stationary distribution is the true posterior, run it for many steps, and treat the resulting samples as draws from $p$. As the number of samples grows, MCMC's answer converges to the exact posterior — it has no representational bias — but each sample is expensive to obtain (an evaluation of the model, often on the whole dataset) and consecutive samples are correlated, so many are needed before the chain has explored the full shape.

Worked example 1: a two-peaked posterior

Suppose the true posterior over a parameter has two roughly equal peaks, at $\theta \approx -3$ and $\theta \approx 3$ (say, from a model with a sign ambiguity). A mean-field Gaussian VI fit, forced to pick one bell curve, settles on a single peak — it might converge to $q = \mathcal{N}(3, 0.5)$ and simply never represent the peak at $-3$ at all, understating the true uncertainty by ignoring half the probability mass. An MCMC chain run long enough, by contrast, spends roughly half its samples near $-3$ and half near $3$, correctly reporting a bimodal posterior. This is the classic VI failure mode: mode-seeking behavior that silently discards other modes.

Worked example 2: the cost trade at scale

Consider a model with $D = 50$ parameters fit to $N = 10{,}000$ data points. Each full-data likelihood evaluation costs roughly $O(ND) = 500{,}000$ operations. MCMC needs enough steps to explore the posterior and de-correlate — say 10,000 iterations, each requiring one full evaluation: $10{,}000 \times 500{,}000 = 5$ billion operations. VI instead runs gradient ascent on the ELBO for, say, 200 iterations to convergence: $200 \times 500{,}000 = 100$ million operations — 50 times cheaper here, at the cost of only ever returning the closest Gaussian-family approximation rather than the exact shape.

A single bell curve like this is the entire family VI is allowed to pick from under a mean-field Gaussian approximation — no matter how it tunes mean and spread, it can never produce two separate humps.

What this means in practice

Production systems that refit a model daily on huge datasets (recommender systems, large Bayesian neural networks) almost always use VI or one of its stochastic variants, because MCMC's per-step cost is too high to repeat on that schedule. Research settings validating a small, critical model — where getting the shape of uncertainty right matters more than speed — lean on MCMC, sometimes using a cheap VI fit only to initialize the MCMC chain closer to the posterior's mass.