Heteroscedastic Regression With Learned Variance

Ordinary regression trained with mean-squared error implicitly assumes every prediction is wrong by roughly the same amount — one error bar, applied everywhere, regardless of whether the input is a calm, predictable region of the data or a genuinely chaotic one. That assumption is usually false: a volatility forecast on a quiet blue-chip stock is naturally more precise than the same forecast on an earnings day for a small-cap. Heteroscedastic regression fixes this by having the network predict its own error bar, input by input, rather than living with one fixed number for every prediction.

The analogy: a forecaster who states their own confidence

A weak weather forecaster always says "expect 20°C" with the same implied confidence, whether it's a calm summer day or the middle of an unpredictable storm system. A better forecaster says "20°C, plus or minus 1 degree" on the calm day, and "20°C, plus or minus 8 degrees" during the storm — the same point forecast, but honest about how much less trustworthy it is under the second set of conditions. A heteroscedastic network is trained to be that second forecaster: it outputs both a mean $\mu(x)$ and a spread $\sigma(x)$ for every input $x$, and $\sigma(x)$ is allowed to be large or small depending on how genuinely noisy that region of the input space is.

The loss that rewards honesty

Training uses the Gaussian negative log-likelihood instead of plain squared error:

$$\mathcal{L} = \frac{1}{2}\log\sigma(x)^2 + \frac{(y - \mu(x))^2}{2\sigma(x)^2}$$

The second term is a squared-error term, but divided by the predicted variance — a prediction that's off by a lot costs less if the network honestly predicted a large $\sigma$ there, and costs a lot if the network claimed high confidence (small $\sigma$) and was wrong. The first term, $\frac12\log\sigma^2$, stops the network from gaming this by simply predicting enormous $\sigma$ everywhere to make every error look cheap — that first term grows whenever $\sigma$ grows, so inflating variance without genuine cause is itself penalized. In plain English: the loss rewards a network for being both accurate and correctly calibrated about how accurate it is.

Worked example 1: comparing a calibrated guess to an overconfident one

Two predictions for the same true value $y = 10$. Prediction A: $\mu=9.8$, $\sigma=0.5$ (small, honest error, modest confidence). Prediction B: $\mu=9.8$, $\sigma=0.05$ (same error, but wildly overconfident). Loss A: $\frac12\log(0.25) + \frac{0.04}{2(0.25)} = -0.693 + 0.08 = -0.613$. Loss B: $\frac12\log(0.0025) + \frac{0.04}{2(0.0025)} = -3.00 + 8.0 = 5.00$. Prediction B's tiny error is amplified into a huge loss because it claimed near-certainty; prediction A, admitting more uncertainty for the same error, is penalized far less. This is exactly the incentive structure that trains the network to widen $\sigma$ honestly rather than bluff confidence.

Worked example 2: two regions with different noise levels

Region A (calm) has residuals typically near $\pm0.2$; region B (volatile) has residuals typically near $\pm2.0$. A network forced to use one shared $\sigma$ for both regions must compromise — say $\sigma=1.1$ — which is too wide for region A (unnecessarily inflating its loss via the log term relative to its tiny true errors) and too narrow for region B (heavily penalizing it via the squared-error term for genuinely large, unavoidable errors). Letting $\sigma(x)$ vary lets the network settle near $\sigma_A \approx 0.2$ and $\sigma_B \approx 2.0$ instead, and the total loss summed across both regions is lower than any single shared value could achieve — the model is rewarded for admitting region B is inherently noisier.

The squared-error term in the loss has exactly this shape before being divided by $\sigma^2$ — heteroscedastic regression's whole trick is letting that division rescale the curve differently in different regions of the input.

What this means in practice

This is the standard way to get input-dependent (aleatoric) uncertainty out of a single deterministic forward pass — no repeated sampling needed, unlike MC dropout — making it cheap enough for real-time risk sizing: a position-sizing model can shrink size automatically on inputs where its own predicted $\sigma(x)$ is wide, without any separate uncertainty-estimation pipeline.