Statistical Decision Theory and Loss Functions

Why is the sample mean the "best" estimator of a population mean, and the sample median sometimes better instead? The honest answer is: it depends what you mean by "best," and "best" is a choice you make, not a fact handed down by mathematics. Statistical decision theory forces that choice into the open by requiring you to specify, in advance, exactly what a wrong estimate costs you — and then it picks the estimator that minimizes that cost. Change the cost, and the "best" estimator changes with it.

An analogy: a doctor choosing which mistake to fear more

A doctor screening for a disease faces two kinds of mistakes: telling a healthy patient they're sick (a false alarm — costly in worry and unnecessary treatment) and telling a sick patient they're healthy (a missed diagnosis — costly in delayed treatment, potentially fatal). No single "correct" screening threshold exists in the abstract; the right threshold depends entirely on how much worse a missed diagnosis is than a false alarm. A doctor screening for a highly treatable, low-risk condition sets the threshold one way; a doctor screening for something fast and lethal sets it very differently, using the same statistical test but a different cost trade-off. Decision theory is that trade-off made explicit and mathematical, applied to any estimation problem, not just diagnosis.

The framework, one symbol at a time

Let $\theta$ be the true parameter and $\hat\theta$ your estimate. A loss function $L(\theta, \hat\theta)$ assigns a number — the cost — to using estimate $\hat\theta$ when the truth is $\theta$. Two standard choices: squared error, $L(\theta,\hat\theta) = (\theta - \hat\theta)^2$, which punishes big misses disproportionately harder than small ones; and absolute error, $L(\theta,\hat\theta) = |\theta - \hat\theta|$, which punishes misses proportionally to their size.

Since $\theta$ is unknown, you can't compute the loss directly — instead you evaluate an estimator by its expected loss, averaged over the randomness in the data, called the risk:

$$R(\theta, \hat\theta) = E_\theta\big[L(\theta, \hat\theta(X))\big]$$

Read this as: "if the truth really is $\theta$, and you repeated the experiment many times, what is the average cost your estimator would rack up?" In plain English: risk is the expected price you pay for using this estimator, given a particular truth — and a "good" estimator is one whose risk is low across the range of truths you consider plausible. Crucially, which estimator has the lowest risk depends entirely on which loss function you picked in the first place.

Picture the estimator's error, $\hat\theta - \theta$, as a distribution like the bell curve above. Squared-error loss weighs the tails of that distribution (the big misses) far more heavily than absolute-error loss does — the same error distribution produces a different risk number, and can favor a different estimator, purely based on which loss function is applied to it.

Compare the curve above — steep, ever-accelerating away from its minimum — to an imagined straight-V shape instead. The quadratic curve is what squared-error loss looks like as a function of the error size; a straight-V is what absolute-error loss looks like. The quadratic punishes a doubling of the error four times as hard; the V only punishes it twice as hard — that difference in steepness is the entire reason the two losses pick different "best" estimators.

Worked example 1: mean vs. median under two different losses

You have three plausible forecasts for tomorrow's return: $1\%, 2\%, 9\%$ (the last one an unusual scenario with a fat-tail spike). You must report a single summary number.

Under squared-error loss, the risk-minimizing choice is the mean: $\bar x = \frac{1+2+9}{3} = 4.0\%$. Check it beats other candidates: sum of squared errors from $4.0$ is $(1-4)^2+(2-4)^2+(9-4)^2 = 9+4+25=38$. From the median, $2.0\%$: $(1-2)^2+(2-2)^2+(9-2)^2 = 1+0+49=50$. The mean wins under squared loss, $38 < 50$.

Under absolute-error loss, the risk-minimizing choice is the median: $2.0\%$. Sum of absolute errors from $2.0$: $|1-2|+|2-2|+|9-2| = 1+0+7=8$. From the mean, $4.0$: $|1-4|+|2-4|+|9-4|=3+2+5=10$. The median wins under absolute loss, $8 < 10$. Same three numbers, opposite "best" answer, purely from the loss function choice.

Worked example 2: an asymmetric loss for a one-sided risk

A trader estimating a security's fair value cares more about overestimating it (buying too high) than underestimating it (missing a cheap entry, a smaller cost). Define an asymmetric loss: $L(\theta,\hat\theta) = 2(\hat\theta-\theta)$ if $\hat\theta > \theta$ (overestimate, penalized double), and $L(\theta,\hat\theta) = (\theta-\hat\theta)$ if $\hat\theta \le \theta$ (underestimate, penalized normally).

Suppose the true value could be $\theta = 98$ or $\theta=102$, each with probability 0.5, and you must pick one fixed estimate $\hat\theta$ from $\{98, 100, 102\}$. Evaluate the risk of $\hat\theta=100$: if $\theta=98$, overestimate by 2, loss $=2\times2=4$; if $\theta=102$, underestimate by 2, loss $=2$. Risk $=0.5(4)+0.5(2)=3.0$. Evaluate $\hat\theta=98$: if $\theta=98$, loss 0; if $\theta=102$, underestimate by 4, loss $=4$. Risk $=0.5(0)+0.5(4)=2.0$ — lower than $100$'s risk of $3.0$. The asymmetric loss pulls the "best" estimate systematically below the naive midpoint, exactly reflecting the trader's stated preference to err cheap rather than err expensive.

What this means in practice

• Portfolio risk models routinely use asymmetric losses, penalizing downside surprises more than upside ones — Value-at-Risk and expected-shortfall-based portfolio construction are decision-theoretic choices, not neutral statistical defaults, and using ordinary squared-error loss (as in plain mean-variance optimization) silently assumes gains and losses matter equally, which is rarely true for a risk manager.
• Choosing "mean forecast" vs. "median forecast" for a P&L projection is itself a decision-theoretic choice — median forecasts are more robust to the occasional extreme scenario, mean forecasts better reflect the true expected dollar outcome, and picking between them should be a conscious decision, not a default.
• The James-Stein estimator and other shrinkage methods are decision-theoretic results: they show that under squared-error loss over several parameters simultaneously, shrinking every individual estimate toward a common target lowers total risk versus estimating each one separately — a genuinely surprising result decision theory makes provable.