Bayes by Backprop

An ordinary neural network learns a single best value for every weight, so two different runs of the same architecture on the same data can converge to different weights, yet both report predictions with no hint of how confident they should be. Bayes by Backprop instead assigns each weight a full probability distribution — typically a Gaussian with its own learned mean and standard deviation — so the network's uncertainty about which weight values are correct becomes an explicit, trainable quantity rather than something thrown away after training.

The obstacle is that you can't backpropagate a gradient through a random sample in the ordinary way, since sampling isn't a differentiable operation. Bayes by Backprop solves this with the reparameterization trick: instead of directly sampling a weight from its distribution, you sample a fixed, weight-independent noise value $\epsilon$ from a standard normal, then compute the weight as $w = \mu + \sigma \cdot \epsilon$, where $\mu$ and $\sigma$ are the trainable mean and standard deviation. Because $\mu$ and $\sigma$ now appear as ordinary multiplied and added terms rather than parameters of a random sampling step, gradients flow through them via standard backpropagation, exactly as they would for any deterministic weight.

Training minimizes a loss balancing two things: how well the network fits the data (using sampled weights), and how far the learned weight distributions have drifted from a chosen prior, via a KL-divergence penalty. In practice, once trained, running the same input through the network multiple times — each time sampling fresh weights from their learned distributions — produces a spread of predictions whose width is a genuine estimate of the model's uncertainty, higher on inputs unlike anything in the training data.