Huber and Robust Loss Functions

Train a model to predict next-quarter earnings growth with ordinary squared-error loss, and one restated, one-off accounting write-down in the training set can quietly rewrite the whole fit. Squared error punishes a residual of 50 with four times the weight of a residual of 25 — so the model bends itself out of shape chasing that single point at the expense of everything else.

Absolute-error loss fixes the domination problem: a residual twice as large only counts twice as much, not four times. But it has a kink at zero, so gradient-based optimizers stall and jitter right where most residuals actually live. The Huber loss is the compromise: quadratic like squared error for small residuals, where you want a smooth gradient, and linear like absolute error for large ones, where you want a damped penalty.

The formula, one piece at a time

Let $r$ be a residual (prediction minus actual). The Huber loss is:

$$L_\delta(r) = \begin{cases} \dfrac{1}{2} r^2 & |r| \le \delta \\ \delta\left(|r| - \dfrac{1}{2}\delta\right) & |r| > \delta \end{cases}$$

In words: if the error is within $\delta$ of zero, square it and halve it, exactly like ordinary least squares. If it's bigger than $\delta$, switch to a straight line whose slope matches the quadratic piece at the seam, so the function is smooth with no kink at $r = \delta$ and no cliff at $r = 0$. The constant $-\frac12\delta^2$ term just keeps the two pieces meeting continuously.

Worked example

Suppose a return-forecasting model is scored with $\delta = 2$ (in percentage points), and it has two residuals in the training set: a routine miss of 1.0 and a wild miss of 8.0 caused by a merger announcement no feature could have seen coming.

• Squared error. Routine miss: $0.5 \times 1.0^2 = 0.5$. Wild miss: $0.5 \times 8.0^2 = 32$. The wild miss contributes 64 times more to the total loss than the routine one.
• Huber loss, $\delta=2$. Routine miss (within $\delta$): $0.5 \times 1.0^2 = 0.5$, unchanged. Wild miss (beyond $\delta$): $2 \times (8.0 - 0.5 \times 2) = 2 \times 7.0 = 14$. The wild miss still costs more, but only 28 times as much, not 64 — its influence on the gradient is capped rather than squared.

What this means in practice

Return data is full of one-off events — earnings surprises, halts, thin-liquidity prints — that are real but not representative. A model trained with squared error effectively lets its gradient step be dictated by whichever training example currently has the worst residual, which drifts as fitting proceeds and can make optimization unstable. Gradient-boosted tree libraries (XGBoost, LightGBM) expose Huber-style objectives directly, and it is a common substitute for squared error whenever a target — returns, P&L, order-fill slippage — has heavy tails.