Model Capacity and Complexity Control

Give a tailor a single piece of stiff cardboard and ask them to fit it to every customer's back — it can only bend into a few shapes, so it fits nobody perfectly, but it also can't be distorted by a customer standing awkwardly for a moment. Give the same tailor a sheet of loose, flexible fabric and it drapes around anyone exactly, wrinkles and all — but a fidgeting customer gets a garment cut to that fidget, not their actual shape. Model capacity is that trade-off: how much a model is able to bend to fit whatever data it's shown, including the parts that are just noise.

What capacity means precisely

Capacity is a measure of how rich the family of functions a model can represent is — how many different input-output patterns it could, in principle, fit exactly. A straight line has low capacity: given any three points, it usually cannot pass through all of them exactly. A degree-20 polynomial has enormous capacity: given 20 points, it can be made to pass through every single one, no matter how those points were generated. A model's capacity is a property of its architecture — the number of adjustable parameters, the depth of a tree, the degree of a polynomial — set before training ever sees the data.

The practical danger is that a model's capacity has nothing to do with how much real signal is present in the data it's given. A degree-20 polynomial fit to 20 noisy points will pass through every single one exactly — training error of zero — while representing almost pure noise, since the true underlying relationship might be close to a straight line. Complexity control is the deliberate act of reining capacity in so the model's flexibility roughly matches how much genuine structure the data actually contains, rather than how many parameters happen to be available to soak up noise.

$$\hat{f} = \arg\min_{f \in \mathcal{F}} \; \frac{1}{n}\sum_{i=1}^n L(y_i, f(x_i)) + \lambda \, \Omega(f)$$

In words: instead of just picking whichever function $f$, from the available family $\mathcal{F}$, best fits the training data (the first term), add a penalty $\Omega(f)$ that grows the more complex $f$ is — larger coefficients, more splits, more nonzero weights — scaled by a strength $\lambda$. A higher $\lambda$ forces the search toward simpler functions even if they fit the training data slightly worse, trading away some training accuracy in exchange for resisting the temptation to fit noise. $\Omega(f)$ is the complexity control; $\lambda$ is how hard it's applied.

Drag model complexity here and watch training error fall monotonically while test error falls, bottoms out, and then rises again. That rising right-hand side is capacity outrunning the available signal — the model is now expressive enough to fit noise specific to the training sample, and that noise doesn't generalize. The complexity level at the test-error minimum is, informally, the "right" amount of capacity for this amount of data.

Worked example 1: polynomial degree versus data size

Fit polynomials of increasing degree to 10 noisy points generated from a true quadratic relationship. Degree 1 (a line) gets training MSE $\approx4.2$ — too rigid to capture the curvature, so both training and held-out error are mediocre. Degree 2, matching the true shape, gets training MSE $\approx1.1$ and held-out MSE $\approx1.3$ — close together, meaning the fit generalizes. Degree 9 gets training MSE $\approx0.0$, a seemingly perfect fit, but held-out MSE explodes to $\approx38$, because the polynomial contorts itself through the specific noise in these 10 points, and that contortion has no relationship to the noise a fresh sample will have.

Worked example 2: the same capacity, controlled two different ways

Two teams both use a degree-9 polynomial (high raw capacity) on the same 10 points from the previous example. Team A fits it with no penalty at all and gets the disastrous held-out MSE $\approx 38$ from above. Team B fits the identical degree-9 family but adds a ridge-style penalty $\lambda \sum \beta_j^2$ with $\lambda = 2.0$, which shrinks the higher-degree coefficients toward zero unless the data strongly demands them. Team B's fitted curve ends up looking almost like the degree-2 fit — the penalty effectively "turns off" most of the extra capacity — and achieves held-out MSE $\approx 1.4$, nearly matching the correctly-specified quadratic model despite starting from a family nine degrees too flexible. The raw capacity of the model family didn't change between A and B; only the complexity control did, and that alone was the entire difference between a 38 and a 1.4.

What this means in practice

In quant research, model capacity has to be chosen relative to how much independent data is actually available — and daily financial data has far fewer truly independent observations than its row count suggests, because adjacent days are correlated. This is exactly why regularized linear and shallow tree-based models remain competitive with deep learning on many trading signals despite deep learning's far higher raw capacity: the extra capacity has nothing genuinely new to explain once the available independent signal is used up, and only additional noise to overfit.