The Mean-Field Approximation

Exact Bayesian posteriors for realistic models are usually a tangled, high-dimensional mess with no closed form, and sampling from them with MCMC can be slow. Variational inference takes a different approach entirely: pick a family of simple, tractable distributions, and search within that family for the member that best approximates the true posterior. The mean-field approximation is the simplest, most common choice of family — it assumes every variable is independent of every other, turning one impossibly complex joint posterior into a product of separate, manageable pieces that can each be optimized on their own.

The analogy: describing a crowd by describing each person alone

Imagine trying to precisely describe how a large crowd of people move together — every subtle correlation, every person nudging their neighbor, every group moving in loose coordination. That joint description is enormously complex. The mean-field approximation is like giving up on capturing any of those correlations, and instead describing each person's movement completely independently, as if no one in the crowd were influenced by anyone else at all. It is obviously a simplification — real crowds do have correlated motion — but it is vastly easier to write down, and if the correlations are weak enough (or if you only need marginal behavior, not joint behavior), it can still be a genuinely useful approximation.

The formula: factorizing the posterior by assumption

The mean-field family assumes the approximate posterior $q$ over all latent variables $z_1, \dots, z_n$ factorizes completely:

$$q(z_1, \dots, z_n) = \prod_{i=1}^n q_i(z_i)$$

In words: instead of allowing $z_1$'s distribution to depend on what value $z_2$ took (or any other variable), the approximation treats every variable as having its own separate, independent distribution $q_i$, and the joint approximate posterior is just the product of these independent pieces. Fitting the approximation means choosing each $q_i$ to make the whole product $q$ as close as possible to the true posterior $p(z\mid x)$, measured by KL divergence — and it turns out (via the calculus of variations) that the optimal $q_i$, holding all the others fixed, has a clean closed form:

$$\ln q_i^*(z_i) = \mathbb{E}_{j\neq i}\big[\ln p(z, x)\big] + \text{const}$$

In words: the best distribution for variable $i$, given everything else's current approximation, is proportional to the expected log joint probability, averaged over all the other variables' current distributions — each $q_i$ is updated one at a time, using the current guesses for everyone else, in a loop called coordinate ascent variational inference (CAVI).

Worked example 1: a two-variable toy model by hand

Take two variables, $z_1$ (a mean) and $z_2$ (a precision, i.e. inverse variance), jointly governing observed data, where the true posterior has $z_1$ and $z_2$ genuinely correlated (as is typical when both are unknown simultaneously). The mean-field approximation forces $q(z_1, z_2) = q_1(z_1)\,q_2(z_2)$ — no correlation allowed at all. Suppose CAVI converges to $q_1 = \text{Normal}(5.0, 0.3^2)$ and $q_2 = \text{Gamma}(4, 2)$ (mean $2.0$). These marginal shapes can be quite accurate individually — the mean-field posterior's mean for $z_1$ might land within a few percent of the true posterior's mean for $z_1$. What is lost entirely is the relationship between them: the true posterior might show that when $z_1$ is unusually large, $z_2$ tends to be unusually small too — that joint tendency simply cannot be represented once you have committed to $q_1$ and $q_2$ being independent by construction.

Worked example 2: underestimating uncertainty

Suppose the true joint posterior variance of $z_1$, accounting for its correlation with $z_2$, is $0.5^2 = 0.25$, but because the correlation happens to work in a variance-reducing direction once $z_2$ is marginalized out incorrectly, the mean-field fit converges to a narrower $q_1 = \text{Normal}(5.0, 0.3^2)$, variance $0.09$. This is a well-known systematic tendency of mean-field approximations: because $q$ cannot represent the correlation structure that would otherwise let uncertainty in one variable "borrow" width from uncertainty in another, mean-field posteriors are frequently overconfident — reporting tighter credible intervals than the truth warrants. A risk model built on this approximation might report a 90% credible interval of roughly $[4.51, 5.49]$ when the true interval, accounting for correlation, is closer to $[4.18, 5.82]$ — a real, quantifiable understatement of genuine uncertainty.

Each factor $q_i$ in a mean-field approximation is a simple distribution like the one above, fit on its own — the whole approximation is just several such curves multiplied together, with no mechanism to represent any curve leaning on another.

What this means in practice

Mean-field variational inference is used wherever exact or MCMC-based posteriors are too slow — large-scale topic models, big hierarchical models with thousands of groups, or any setting needing repeated, fast posterior updates (like real-time model retraining). Because it tends to underestimate posterior uncertainty and can also settle on a single mode when the true posterior is multi-modal, it should be treated as a fast approximation to check against, not an automatic substitute for MCMC when calibrated uncertainty genuinely matters — for instance, in a risk model where understating tail uncertainty has direct financial consequences.

Practice

1. Explain in one sentence why forcing $q(z_1,z_2) = q_1(z_1)q_2(z_2)$ cannot represent a true posterior where $z_1$ and $z_2$ are correlated.
2. If two variables in a model are known to be nearly uncorrelated in the true posterior, would you expect mean-field to perform well or poorly? Why?
3. Name one financial modeling context where understating posterior uncertainty (as mean-field tends to do) would be especially costly.