The Laplace Approximation for Neural Networks

Training a neural network from scratch to output calibrated uncertainty — as methods like Bayes by Backprop do — requires changing the training procedure entirely. The Laplace approximation takes a different, much cheaper route: train an ordinary neural network exactly as usual to get one set of best-fit weights, then, only after training is finished, approximate the shape of the loss surface around those weights with a Gaussian.

The intuition is that near the trained weights, the loss surface curves upward in every direction (it's a minimum), and how sharply it curves tells you how much you could nudge each weight without hurting performance much. A direction where the loss curves very steeply means that weight is tightly pinned down by the data — little uncertainty. A direction where the loss is nearly flat means many different weight values fit almost equally well — high uncertainty. The Laplace approximation captures this by computing the Hessian (matrix of second derivatives) of the loss at the trained weights, and using its inverse as the covariance of a Gaussian distribution centered on the trained weights.

Because computing and storing the full Hessian for a network with millions of weights is computationally infeasible, practical implementations approximate it — commonly using only the diagonal, or a block-diagonal structure per layer — trading some accuracy in the uncertainty estimate for tractability. The appeal is that this whole procedure is a post-hoc add-on: any already-trained network, even one someone else trained, can get an uncertainty estimate bolted onto it without retraining, at a fraction of the computational cost of a fully Bayesian training method.