Loss Functions for Regression

Every regression model needs a number that says "how wrong was this prediction, in total, across all the data points." Without that single number there is nothing to minimise and no way to compare one model to another. The loss function is that number's recipe, and the recipe you pick quietly decides what kind of model you end up with.

The analogy: grading a trader's forecasts

Imagine grading a trader's next-day price forecasts against what actually happened. One scheme: penalty equals the squared size of the miss. A $1 miss costs 1 point, a $10 miss costs 100 points. A different scheme: penalty equals the plain size of the miss — a $1 miss costs 1 point, a $10 miss costs 10 points. Both schemes agree on small misses. They violently disagree on big ones. The first scheme treats one $10 miss as worse than ten separate $1 misses combined; the second treats them as equal. Which scheme you use changes what "the best forecaster" means, and it changes what a model trained to minimise that score will actually do.

Squared error: punish big misses hardest

The most common loss, mean squared error, is

$$\text{MSE} = \frac{1}{n}\sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

Here $y_i$ is the true value, $\hat y_i$ the model's prediction, and $n$ the number of data points. In words: take every miss, square it so sign doesn't matter and big misses count extra, then average. Because the penalty grows with the square of the miss, a model trained to minimise MSE will bend over backwards to avoid even one huge error, at the cost of tolerating many small ones. That is exactly what makes it fragile: a handful of outliers — a fat-fingered print, a data error — can drag the whole fit toward them.

Mean absolute error: every dollar the same

$$\text{MAE} = \frac{1}{n}\sum_{i=1}^{n} |y_i - \hat{y}_i|$$

Same setup, but the penalty is the plain size of the miss, not its square. A $100 miss costs exactly a hundred times a $1 miss — no extra penalty for being big. MAE is minimised by the median of the residuals rather than the mean, which is why it barely moves when a few outliers show up: the median doesn't care how far outliers are, only that they're on one side or the other.

Worked example 1: five residuals, two rulers

Residuals (true minus predicted) for five days: $1, -1, 2, -2, 10$. The last one is a bad miss — maybe a data glitch.

MSE: square each, average. $1 + 1 + 4 + 4 + 100 = 110$, divided by 5 gives $\text{MSE} = 22$.

MAE: take absolute values, average. $1+1+2+2+10 = 16$, divided by 5 gives $\text{MAE} = 3.2$.

Drop the outlier and use only the first four residuals: $\text{MSE} = 2.5$, $\text{MAE} = 1.5$. The single outlier multiplied MSE by nearly 9× but MAE by only about 2×. That gap is the entire practical difference between the two losses.

Worked example 2: Huber loss, the compromise

Huber loss acts like squared error for small misses and like absolute error for large ones, switching at a threshold $\delta$:

$$L_\delta(r) = \begin{cases} \tfrac{1}{2}r^2 & |r| \le \delta \\ \delta\left(|r| - \tfrac{1}{2}\delta\right) & |r| > \delta \end{cases}$$

where $r = y - \hat y$ is the residual. Set $\delta = 2$. For $r = 1$ (small miss, use the top branch): $L = 0.5(1)^2 = 0.5$. For $r = 10$ (large miss, use the bottom branch): $L = 2(10 - 1) = 18$. Compare to squared error's $100$ for the same miss — Huber caps the damage a single bad point can do while still behaving like ordinary least squares near zero.

Try dragging the loss surface's steepness in the explorer above — a squared-error surface curves smoothly to its minimum, which is why gradient descent on MSE converges predictably, while a surface built from absolute error has a kink at zero and needs a slightly different treatment.

Compare the quadratic shape here to a straight-line (absolute-value) penalty in your head: the quadratic grows much faster away from zero, which is the picture behind "squared error punishes big misses hardest."

What this means in practice

A quant forecasting daily returns from noisy, glitch-prone data usually prefers MAE or Huber loss over plain MSE, because a single bad tick shouldn't be allowed to warp the whole model. Conversely, if the cost of an error genuinely does grow quadratically — say, a hedging error whose P&L impact scales with the square of the mis-hedge — then MSE is the correct choice, not just the default one. The loss should match the real economic cost of being wrong, not just be whatever the library defaults to.