Bayesian Neural Networks

A standard neural network learns one specific value for every weight, and gives you one specific answer for a new input — with no indication of whether that's a well-supported prediction or a coin flip it happens to have committed to. An input from the middle of the training data and one wildly out of distribution can produce equally confident-looking predictions, because the architecture has no built-in way to say "I've never seen anything like this."

A Bayesian neural network treats every weight not as one fixed number but as a range of plausible values, weighted by how consistent each value is with the training data — the same logic as an analyst who says "the fair value is probably around $52, most likely somewhere between $48 and $56, and I'd bet heavily against it being outside $40–$65" instead of just stating "$52" with no sense of how firm that number is. When the model has seen plenty of relevant data, that range of plausible weights is narrow and predictions are tight; when the input looks unfamiliar, the range widens and the model's own uncertainty grows along with it.

The mechanics, one symbol at a time

Standard training finds a single best weight vector $w$ by minimizing a loss. Bayesian training instead computes a full distribution over $w$ given the data $D$, using Bayes' rule:

$$p(w \mid D) = \frac{p(D \mid w)\, p(w)}{p(D)}$$

In words: the posterior $p(w \mid D)$ — what you believe about the weights after seeing the data — is proportional to how well weights $w$ explain the data (the likelihood, $p(D \mid w)$) times what you believed about the weights before seeing any data (the prior, $p(w)$, typically a simple assumption like "small weights are more plausible than huge ones"). $p(D)$ is just a normalizing constant that makes probabilities sum to 1 and doesn't change which weights are more or less plausible relative to each other.

Computing this exactly is intractable for any network with more than a handful of weights, so in practice it's approximated — commonly with variational inference, which fits a simpler, tractable distribution (like an independent Gaussian per weight, with a learned mean and spread) to be as close as possible to the true posterior. A prediction is then made by sampling several different weight settings from that approximate posterior, running the network forward with each one, and looking at how much the resulting predictions agree or disagree.

Worked example 1: uncertainty from disagreement across sampled weights

A Bayesian network's approximate posterior for a single output weight is $\mathcal{N}(\mu=0.8, \sigma=0.05)$ — a fairly narrow distribution, reflecting confidence built from lots of relevant training data. Draw three samples: $w_1 = 0.83$, $w_2 = 0.79$, $w_3 = 0.77$ (all close together, as expected from a small $\sigma$).

For an input with hidden activation $h = 5.0$, the three predictions are $\hat{y}_1 = 0.83(5)=4.15$, $\hat{y}_2=0.79(5)=3.95$, $\hat{y}_3=0.77(5)=3.85$ — a tight spread of roughly $4.15$ to $3.85$, about $\pm 0.15$ around a mean near $4.0$.

Now consider an input the network is far less sure about, where the relevant weight's posterior has widened to $\mathcal{N}(\mu=0.8,\sigma=0.4)$ — say, because this input activates a part of the network rarely exercised by training data. Samples might be $w_1=1.1$, $w_2=0.6$, $w_3=0.9$: predictions $5.5$, $3.0$, $4.5$ — a spread of $2.5$, over 15 times wider. Same input activation, same mean weight, but the disagreement across samples reveals the model's genuine uncertainty about that region of its parameter space.

Worked example 2: how the prior pulls weights toward simplicity

With a Gaussian prior $p(w) = \mathcal{N}(0, 1)$ on a weight, the posterior for a small dataset is pulled noticeably toward zero relative to what an unconstrained fit would give. Suppose the likelihood alone (ignoring the prior — what a standard network would converge to) would place the weight's best estimate at $w_{\text{ML}} = 3.0$, but with only 10 training examples supporting that estimate. Weighing that weak, small-sample evidence against a prior centered at 0 pulls the posterior mean well short of 3.0, say to around $1.2$ — with so little data, the prior dominates. With 10,000 supporting examples instead of 10, the same weighing lands almost exactly at $3.0$: the data overwhelms the prior once there's enough of it. This is the same mechanism as L2 regularization, made explicit: a prior favoring small weights defends against overfitting to a handful of examples.

Standard training (visualized in the explorer above) finds the single point at the bottom of the loss surface. A Bayesian network instead tries to characterize the whole shape of the loss surface near good solutions — a wide, flat bowl means many weight settings fit almost equally well (high uncertainty); a narrow, steep bowl means the data pins the weight down tightly (low uncertainty).

What this means in practice

Bayesian neural networks are used wherever knowing how much to trust a prediction matters as much as the prediction itself — position sizing that shrinks when the model is uncertain, flagging live conditions unlike anything in training data, or risk models where overconfident tails are costly. The practical cost is real: exact Bayesian inference is intractable at neural-network scale, so every deployed method is an approximation (variational inference, or a cheaper proxy like Monte Carlo dropout — running a dropout-trained network multiple times with dropout left on and treating the output spread as an uncertainty estimate), and that approximation's quality determines how trustworthy the resulting numbers are.