The Laplace Approximation

Full MCMC sampling of a posterior distribution is thorough but can be slow — thousands of iterations, careful convergence checking, real computing time. Sometimes you don't need the full picture; you just need a fast, reasonable summary of where the posterior peaks and how spread out it is, to plug into a downstream calculation or get a quick answer before committing to a longer MCMC run. The Laplace approximation is exactly that shortcut: it replaces the true, possibly complicated posterior with the nearest-fitting bell curve, found using nothing more than calculus at a single point.

An analogy: fitting a parabola to the top of a hill

Imagine standing at the very peak of a bumpy hill and wanting to describe the hill's overall shape quickly, without walking its entire perimeter. A reasonable shortcut: measure how steeply the ground curves away from the peak in every direction right where you're standing, and use that local curvature to fit a smooth, symmetric bowl (a parabola) that matches the peak's height and steepness exactly. Far from the peak, the real hill might look nothing like a smooth parabola — it could have a cliff on one side and a gentle slope on the other — but near the top, the parabola is often a decent stand-in, and it's vastly cheaper to compute than surveying the whole hill. The Laplace approximation does exactly this to a posterior distribution's log-density instead of a physical hill.

The maths, one symbol at a time

Let $\theta^*$ be the posterior mode — the value of $\theta$ that maximizes $\log p(\theta \mid D)$ (found the same way as any optimization, e.g. by gradient ascent). The Laplace approximation fits a normal distribution centered there:

$$p(\theta \mid D) \approx N\!\left(\theta^*,\ \left[-\nabla^2 \log p(\theta^* \mid D)\right]^{-1}\right)$$

In words: approximate the true posterior with a bell curve centered exactly at its peak $\theta^*$, whose width is set by how sharply curved the log-posterior is at that peak — the second derivative (curvature), $\nabla^2\log p$, measured right at the top. A sharply peaked log-posterior (steep curvature) means the true posterior is tightly concentrated, so the approximating normal is narrow; a gently curved log-posterior means the true posterior is diffuse, so the approximating normal is wide. This requires finding only one point (the mode, via optimization) and one local calculation (the curvature there) — dramatically cheaper than exploring the whole distribution via MCMC.

Worked example 1: approximating a Beta posterior with a normal

A win rate has posterior Beta$(20, 10)$ (after some trades), with true mode at $\theta^* = (20-1)/(20+10-2) = 19/28 \approx 0.679$. Computing the curvature of $\log p(\theta)$ at this mode gives an approximating normal with mean $0.679$ and standard deviation approximately $0.083$. Comparing the true Beta$(20,10)$ density to this Laplace-approximated $N(0.679, 0.083^2)$: they agree closely near the peak and are both reasonably symmetric here, since a Beta with these parameters is roughly bell-shaped away from the boundaries — the approximation works well in this regime.

Worked example 2: where the approximation breaks down

Now consider a posterior for a win rate after just 2 trades, both wins: Beta$(2,1)$. Its true mode is at $\theta^* = (2-1)/(2+1-2) = 1$, sitting right at the boundary of the allowed range $[0,1]$, and the true density is sharply asymmetric — it rises steeply toward 1 with no room to spread on the right-hand side at all, since $\theta$ can't exceed 1. A Laplace approximation naively fit here would center a symmetric bell curve at $\theta^*=1$ and assign meaningful probability to values above 1 — a nonsensical result for a probability, and a visibly bad approximation of a distribution that's nothing like bell-shaped this close to a boundary with so little data.

Compare the smooth, symmetric shape in the explorer above to a skewed real posterior near a boundary — the gap between them is exactly the error a Laplace approximation introduces whenever the true posterior isn't genuinely bell-shaped.

What this means in practice

The Laplace approximation is used for fast, "good enough" posterior summaries when full MCMC is too slow — quick model prototyping, computing an approximate marginal likelihood for model comparison (see marginal likelihood), or as an initialization/sanity-check step before committing to a longer MCMC run. It's also the backbone of several fast approximate-inference methods used in large-scale Bayesian machine learning, where full MCMC would be computationally infeasible.