SWAG: Stochastic Weight Averaging-Gaussian

When a neural network is trained with a constant or cyclical learning rate late in training, its weights don't settle onto one exact point — they wander in a small region around a good solution, bouncing slightly from one training step to the next because each mini-batch of data pulls the weights in a slightly different direction. SWAG turns this ordinary training noise into an uncertainty estimate instead of treating it as something to be minimized away.

The recipe: run ordinary training as normal, but during the last portion of training, periodically snapshot the network's weights (say, once per epoch). From those snapshots, compute the average weight vector (this is plain stochastic weight averaging, known to also improve accuracy) and, separately, estimate the covariance of how the snapshots varied around that average. SWAG then treats the trained network's weights as if they were drawn from a Gaussian with that mean and covariance — giving a distribution over plausible weight settings built entirely from snapshots collected during standard training, with essentially no extra computation beyond storing a handful of checkpoints.

To make predictions, you sample several weight vectors from this fitted Gaussian, run the network forward once per sampled weight vector, and look at the spread across those outputs — wider disagreement across samples signals higher uncertainty. Unlike the Laplace approximation, which needs the Hessian at a single point, SWAG's covariance comes from genuinely different weight settings the optimizer actually visited during training, capturing some of the natural noise in how gradient descent explores the loss surface rather than a purely local curvature estimate.