Empirical Risk Minimization

You want the model that will be least wrong on cases you have not met yet. That quantity is unmeasurable by construction — the cases have not happened. So every practical fitting procedure makes the same substitution: score candidate models on the data you do have, and pick the winner. That substitution is empirical risk minimization, or ERM, and it is the engine inside ordinary least squares, logistic regression, support vector machines, boosted trees and neural networks alike.

The everyday version is choosing a restaurant from its reviews. What you care about is how good your dinner will be. What you can measure is the average of a few hundred dinners other people already had. The average is not the thing you want, but it is evidence about the thing you want, and with enough reviews it is good evidence. With three reviews it is not.

The two risks

The thing you want is the true risk — the expected loss over the whole data-generating process:

$$R(f) = \mathbb{E}_{(X,Y)\sim P}\big[\ell(f(X),\, Y)\big]$$

Read it as: how badly rule $f$ does on average, over every case the world could ever produce. The thing you can compute is the empirical risk — the same average, taken over your $n$ rows instead of over the world:

$$\hat{R}_n(f) = \frac{1}{n}\sum_{i=1}^{n} \ell\big(f(x_i),\, y_i\big)$$

Read it as: the average loss the rule racked up on the examples you actually have. ERM is then a single line: search your hypothesis class $\mathcal{H}$ and return whichever rule scores lowest.

$$\hat{f} = \arg\min_{f \in \mathcal{H}} \; \hat{R}_n(f)$$

That is the whole idea. Everything else in this page is about the gap between the two risks.

Worked example 1: two candidate lines

Five observations of a signal $x$ and a return $y$:

$x$	1	2	3	4	5
$y$	2.2	2.8	4.5	4.7	6.3

Use squared loss, $\ell(\hat{y}, y) = (\hat{y}-y)^2$. Candidate A is $f(x) = 1.5 + 0.95x$; candidate B is $f(x) = 2 + 0.8x$.

A predicts $2.45,\, 3.40,\, 4.35,\, 5.30,\, 6.25$. The errors are $-0.25,\, -0.60,\, 0.15,\, -0.60,\, 0.05$; squaring and adding gives $0.0625 + 0.36 + 0.0225 + 0.36 + 0.0025 = 0.8075$, so $\hat{R}_5(A) = 0.8075/5 = 0.162$.

B predicts $2.80,\, 3.60,\, 4.40,\, 5.20,\, 6.00$, with errors $-0.60,\, -0.80,\, 0.10,\, -0.50,\, 0.30$. The squares sum to $1.35$, so $\hat{R}_5(B) = 0.270$.

ERM picks A. And if you let the search run over all lines rather than just these two, you get the least-squares fit $f(x) = 1.07 + 1.01x$ with $\hat{R}_5 = 0.092$ — which is exactly what Ordinary Least Squares (OLS) is: ERM with squared loss over the class of linear functions.

Drag the points below and watch the fitted line chase them. The vertical stubs are the residuals; ERM is the machine that makes the sum of their squares as small as it can.

Worked example 2: picking a threshold

Now a classification problem with 0-1 loss (one unit per mistake). Eight days, each with a signal value and a realised direction:

signal	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8
label	down	down	down	up	up	down	up	up

The class $\mathcal{H}$ is "go long when the signal exceeds $t$", one rule per threshold. Take $t = 0.35$: it calls up on the last five days, which are up, up, down, up, up — one mistake — and down on the first three, all correct. So $\hat{R}_8 = 1/8 = 0.125$. Take $t = 0.55$: it calls up on the last three (down, up, up — one mistake) and down on the first five (three of which were up — two more mistakes), giving $\hat{R}_8 = 3/8 = 0.375$. Take $t = 0.15$: three mistakes above the line, none below, again $0.375$.

ERM returns $t = 0.35$. But notice how little separates the candidates: one flipped day would change the answer. That fragility is the real lesson.

What this means in practice

Almost every named algorithm is ERM with a different loss and a different class. Ordinary Least Squares (OLS) is squared loss over linear functions. Logistic Regression is log loss over linear functions. Support Vector Machines is hinge loss plus a size penalty. Gradient boosting is ERM over sums of trees, solved greedily. Recognising this collapses a dozen "methods" into one idea with knobs.

The fix for the gap is not to optimise harder — it is to shrink what you optimise over. Regularised ERM adds a complexity penalty:

$$\hat{f} = \arg\min_{f \in \mathcal{H}} \; \hat{R}_n(f) + \lambda\,\Omega(f)$$

which says: pay for fit, but also pay for complication, with $\lambda$ setting the exchange rate. That is Ridge and LASSO Regularization, and it is why a slightly worse training score often buys a much better live one.