Hypothesis Space and Inductive Bias

Here is an uncomfortable fact that most people meet only after their first model fails in production. A finite dataset never determines a rule. Whatever numbers you have collected, there are infinitely many different rules that reproduce every one of them exactly and then say wildly different things about the very next case. Not slightly different — arbitrarily different. One says the answer is 5, another says 7, another says minus 40, and the data you hold cannot break the tie because all three agree perfectly with every observation you made.

So a learning algorithm cannot possibly be "letting the data speak". It must be bringing something of its own to the table: a restriction on which rules it will even consider, and a preference ordering over the survivors. Those two things are the hypothesis space and the inductive bias, and together they do at least as much of the work as your data does. Change them and you change the prediction, with the training set untouched.

The analogy: hiring off a shortlist

You are filling a trading seat. Roughly four million people on earth could conceivably do the job. You do not interview four million people. You interview eleven, because your recruiter handed you a shortlist.

That shortlist is the hypothesis space. Everything about the eventual hire was decided in two places: which names made the list, and how you ranked the names on it. If the best possible candidate was never shortlisted, no amount of interviewing recovers them — your process has an upper bound it can never beat. If two candidates interview identically, your tiebreaker decides, and your tiebreaker is a preference you brought with you, not evidence.

Learning works the same way. The shortlist is the set of functions your model class can express — all straight lines, all depth-6 trees, all networks with this architecture. The tiebreaker is everything else: the penalty term, the initial weights, the optimiser, when you stopped training. Both are choices. Both are assumptions about how the world is likely to behave.

Naming the pieces

Write $\mathcal{H}$ (script H) for the hypothesis space — the whole set of functions your algorithm is allowed to return. A single member of that set is written $h$, one hypothesis: a specific rule that turns an input into a prediction. Write $f^{*}$ for the true relationship out in the world, which you never see.

Training picks the member of the shortlist that fits your data best:

$$\hat{h} \;=\; \arg\min_{h \,\in\, \mathcal{H}} \; \frac{1}{n}\sum_{i=1}^{n} \ell\big(h(x_i),\, y_i\big)$$

In plain English: look at every rule on the shortlist, score each one by its average mistake $\ell$ across your $n$ training examples, and return the winner. The symbol $\arg\min$ just means "the thing that achieves the minimum" rather than the minimum value itself.

The trouble is that this line is often satisfied by many rules at once — sometimes by an infinite family of them, all scoring exactly zero. Inductive bias is whatever breaks that tie. It is not a bug and not a bias in the statistical sense of a systematic error. It is the assumption that makes generalisation possible at all.

Worked example 1: two points, three shortlists

Take exactly two observations: at $x = 0$ the answer was $1$, and at $x = 1$ the answer was $3$. Now try three different shortlists.

Shortlist A — constants only. Every rule has the form $h(x) = c$, one number for all inputs. The best $c$ is the average, $c = 2$. Training error is not zero: the squared errors are $(1-2)^2 = 1$ and $(3-2)^2 = 1$, so the mean squared error is $1$. This shortlist is too small to contain anything that fits.

Shortlist B — straight lines. Rules look like $h(x) = a + bx$, where $a$ is the value at zero and $b$ is the slope. Two equations, two unknowns: $a = 1$ from the first point, then $1 + b = 3$ gives $b = 2$. Exactly one rule fits, $h(x) = 1 + 2x$, and it predicts $h(2) = 5$. A unique answer, forced by the shortlist rather than by the data.

Shortlist C — quadratics. Rules look like $h(x) = a + bx + cx^2$, three unknowns against two equations. Again $a = 1$, and the second point now demands $1 + b + c = 3$, so $b = 2 - c$. Every value of $c$ works:

$$h_c(x) \;=\; 1 + (2 - c)\,x + c\,x^2 \qquad \text{for any real } c$$

In words: there is a whole one-parameter family of quadratics through those two points, and every single one has zero training error. Now ask each of them for $x = 2$. Substituting, $h_c(2) = 1 + 2(2-c) + 4c = 5 + 2c$. So $c = 0$ predicts $5$, $c = 1$ predicts $7$, and $c = -2$ predicts $1$. Pick $c = 500$ and the prediction is $1005$. Every one of those answers is perfectly consistent with your data.

The lesson is exact and it is not about noise or small samples. Widen the shortlist and you gain the ability to fit, but you lose the ability to conclude.

Worked example 2: same fit, two tiebreakers, two answers

Now hold the shortlist fixed and change only the tiebreaker. One observation, two features: the input is $x = (x_1, x_2) = (1, 2)$ and the answer was $y = 2$. The model is $h(x) = w_1x_1 + w_2x_2$, where $w_1$ and $w_2$ are the weights on the two features. Fitting perfectly means

$$w_1 + 2w_2 = 2$$

which in words says: any pair of weights sitting on that line reproduces the observation exactly. Infinitely many candidates, all tied on training error. Two common tiebreakers:

Smallest sum of squares (the ridge preference). Minimise $w_1^2 + w_2^2$ along the line. The answer is the point on the line closest to the origin, which sits in the direction $(1,2)$: write $w = t\,(1,2)$, substitute to get $t + 4t = 2$, so $t = 0.4$ and $w = (0.4,\ 0.8)$. Check the fit: $0.4 + 2(0.8) = 2$. Its squared size is $0.16 + 0.64 = 0.80$.

Smallest sum of absolute values (the lasso preference). Minimise $|w_1| + |w_2|$ along the same line. Substituting $w_1 = 2 - 2w_2$ gives a total of $2 - w_2$ for $w_2$ between $0$ and $1$, which keeps falling until $w_2 = 1$. So $w = (0,\ 1)$, with total size $1$. It also fits perfectly: $0 + 2(1) = 2$.

Each tiebreaker prefers its own winner — the ridge answer has absolute size $0.4 + 0.8 = 1.2$, worse than $1$; the lasso answer has squared size $0 + 1 = 1$, worse than $0.80$. Now predict on a new input $x = (2,1)$. Ridge says $0.4(2) + 0.8(1) = 1.6$. Lasso says $0(2) + 1(1) = 1.0$. A 60% difference in the forecast, from identical data and identical training error. The gap is pure inductive bias: ridge assumes influence is shared across correlated features, lasso assumes only a few features matter at all. In a factor model where two signals are near-duplicates, that choice decides your whole portfolio tilt.

Drag the flexibility slider below. Watch the boundary go from a rigid line that cannot bend around the data to a wandering shape that captures every point — the same widening of the shortlist you just did by hand.

What this means in practice

Your model choice is a market view. Fitting a linear factor model is a claim that exposure and return scale proportionally. Fitting a gradient-boosted tree is a claim that the world is made of thresholds and interactions. Neither is neutral, and in a low signal-to-noise setting like returns the assumption does most of the forecasting work, because the data is far too weak to overrule it.

Restriction is what buys you generalisation. A shortlist so wide it fits anything tells you nothing, exactly as in the quadratic example. This is why constraints that look crude — long-only, weight caps, sign restrictions, monotonicity in a factor — so often improve live results while making the backtest look worse. They shrink the shortlist toward rules that were plausible before you saw the data.

When you cannot restrict, prefer. With more parameters than observations you cannot narrow the shortlist enough, so you lean on the tiebreaker: a penalty, an early stop, a shrinkage target, a prior. That is the entire content of Regularization as a Prior.

Bias in, bias out. If your assumption is right you generalise from very little data. If it is wrong you generalise confidently in the wrong direction, and more data will not save you — it will just make you more certain. This is the practical face of the The No Free Lunch Theorem: no shortlist is best everywhere, so the only honest question is whether yours matches this particular market.