Huber Loss for Noisy Financial Targets

Train a neural network to predict next-day returns using squared error, and one data-cleaning slip — a stock split not adjusted for, a fat-finger print, an earnings-day 40% gap — creates a single residual that dwarfs every other example in the dataset. Because squared error squares that residual, one bad target can contribute more to the total loss, and to the gradient, than thousands of ordinary days combined. The optimizer chases that one point and distorts the fit everywhere else. Huber loss is designed specifically to stop this from happening.

The analogy: a car's suspension over a pothole

A stiff suspension gives a precise, responsive ride over small bumps — every little variation in the road is transmitted faithfully. But hit one genuine pothole at speed and that same stiffness slams the whole car. A good suspension is nonlinear: firm and precise for small bumps, softer and more forgiving once the bump crosses a certain size, so one bad pothole doesn't define the whole ride. Huber loss gives a model the same kind of suspension: precise (squared-error-like) for ordinary residuals, and deliberately gentler (absolute-error-like) once a residual exceeds a threshold.

The formula

For a residual $r = y - \hat{y}$ (true value minus prediction) and a threshold $\delta$:

$$L_\delta(r) = \begin{cases} \dfrac{1}{2}r^2 & |r| \le \delta \\[6pt] \delta\left(|r| - \dfrac{1}{2}\delta\right) & |r| > \delta \end{cases}$$

In words: for small residuals (within $\delta$ of the target), use ordinary squared error, which is smooth and gives a gradient proportional to the residual itself. Beyond $\delta$, switch to a term that grows only linearly in $|r|$ rather than quadratically — so a residual of 100 contributes only about 10 times more loss than a residual of 10, not 100 times more. The two pieces are constructed to meet smoothly (same value and same slope) exactly at $r = \delta$.

Worked example 1: a normal-sized error and a huge outlier, $\delta = 1$

Ordinary residual $r = 0.5$ (within the threshold): $L = \tfrac12(0.5)^2 = 0.125$ — same as plain squared error would give.

Outlier residual $r = 20$ (a badly cleaned data point): squared error would give $\tfrac12(20)^2 = 200$. Huber loss instead gives $L = 1 \times (20 - 0.5) = 19.5$ — over 10 times smaller than squared error's penalty, and critically, the gradient at this point is just $\delta = 1$ (constant, the same as at $r=1.5$ or $r=50$), not $r = 20$ as it would be under squared error. The optimizer takes a normal-sized step toward fixing this point instead of a step 20 times larger than usual.

Worked example 2: comparing total loss contribution across a small batch

Batch of 4 residuals: $[0.2, -0.3, 0.5, 15]$ (the last one an outlier), with $\delta = 1$.

Squared error: $\tfrac12(0.04 + 0.09 + 0.25 + 225) = 112.7$ — the outlier alone accounts for $112.5$, or 99.8% of total loss. The other three points are numerically irrelevant to the gradient.

Huber loss ($\delta=1$): the first three, within threshold, contribute $0.02 + 0.045 + 0.125 = 0.19$. The outlier contributes $1 \times (15 - 0.5) = 14.5$. Total $= 14.69$ — the outlier still dominates, but at a ratio of about 76:1 rather than squared error's roughly 590:1, leaving the three normal points a meaningful share of the gradient.

What this means in practice

Huber loss is the standard robust regression choice wherever targets are numeric and noisy — return prediction, volatility forecasting, PnL regression — and is available in every framework (nn.SmoothL1Loss in PyTorch is a close variant). The threshold $\delta$ is a real hyperparameter to tune: set it too small and Huber loss behaves almost entirely like absolute error, discarding useful curvature information from ordinary-sized residuals; set it too large and it behaves almost entirely like squared error, reintroducing exactly the outlier sensitivity it exists to fix.