Parametric vs Nonparametric Models

Ask two architects to design a house. The first works from a fixed blueprint template with a few adjustable numbers — bedroom count, living room width — and once those are set, every other detail follows automatically from the template. The second starts with a blank site and lets the house's shape emerge from the plot's specifics, growing more elaborate the more requirements are added, with no fixed template capping the final complexity. Both can build good houses. The first needs far less information to start but is stuck with the template's shape even if it's wrong for this plot. The second matches any plot exactly but needs to see the full complexity of the situation before the design settles. That's parametric versus nonparametric modelling.

The defining test: does complexity depend on data size?

A parametric model assumes the relationship between inputs and output has a fixed functional form — linear regression, logistic regression, a Gaussian distribution with unknown mean and variance — described entirely by a fixed, finite number of parameters, decided before any data arrives. Linear regression on $p$ features always has exactly $p+1$ coefficients, whether trained on 50 rows or 5 million; more data refines the estimates of those same fixed parameters but never adds new ones.

A nonparametric model makes no such commitment to a fixed shape. A $k$-nearest-neighbours predictor, a decision tree grown to full depth, or a kernel density estimate lets its effective complexity grow as more data arrives — a tree can add more splits, a nearest-neighbour method effectively has "one parameter per training point" in the sense that every stored example can influence a prediction. "Nonparametric" doesn't mean zero parameters; it means the number of parameters isn't fixed in advance and can grow without bound as data grows.

$$\hat{y}(x) = \hat\beta_0 + \hat\beta_1 x \qquad \text{vs.} \qquad \hat{y}(x) = \frac{1}{k}\sum_{i \in N_k(x)} y_i$$

The left form is parametric: two numbers, $\hat\beta_0$ and $\hat\beta_1$, fully determine the prediction at every possible $x$, and no amount of extra data changes how many numbers there are. The right form is $k$-nearest-neighbours: $N_k(x)$ means "the $k$ training points closest to $x$," and the prediction is just their average $y$. There is no fixed formula shape here at all — the prediction is defined entirely by which training points happen to be nearby, and the whole training set is effectively the "parameters."

A logistic curve like this is the entire hypothesis: pick two or three numbers (steepness, midpoint) and the whole function is nailed down everywhere, including regions with no data. A nonparametric method fitting the same data wouldn't commit to any smooth curve shape at all — it would let the fit wiggle to match local clusters of points, and adding more points could change the fitted shape in that region rather than just refining the same two or three numbers.

Worked example 1: fitting the same 6 points two ways

Six points: $(1,2), (2,3), (3,7), (4,7.5), (5,8), (6,15)$. A parametric linear fit finds the single best-fitting line through all six, say $\hat y = 0.5 + 2.1x$, minimizing total squared error — two numbers, $\hat\beta_0=0.5$ and $\hat\beta_1=2.1$, and that's the entire model, whatever new $x$ you ask it about.

A nonparametric 1-nearest-neighbour fit instead just memorizes: predict for $x=3.5$ by finding the closest training point, $x=4$, and returning its $y=7.5$ exactly, or predict for $x=1.5$ by returning the closer of $x=1$ or $x=2$'s value. There are no coefficients to report at all — the "model" is just the six stored points and a rule for finding the nearest one. Where the line smooths straight through the data with a fixed slope, nearest-neighbour reproduces every local bump and jump exactly, including the possibly-noisy jump from $8$ to $15$ between $x=5$ and $x=6$.

Worked example 2: how each behaves as data grows

Suppose the true relationship is a smooth curve with a bend the linear model can't represent — say $y=x^2/2$. With 10 points, the linear fit produces a straight-line compromise, systematically off at the extremes and the middle — a shape mistake no amount of additional data of the same kind fixes, since the model only ever has two numbers. With 10,000 points, those two coefficients become precisely estimated, but the fitted line is still a straight line — precision improved, the shape mismatch didn't.

The nonparametric nearest-neighbour fit with only 10 points is noisy, doing worse than the line in this small-data regime. But with 10,000 points densely covering the range, its local averaging traces out the true curve closely, because there are now enough nearby points everywhere to average away noise while following genuine local curvature — something the two-number line structurally cannot ever do.

What this means in practice

Quant research defaults to parametric models — linear factor models, logistic regression for classification, GARCH for volatility — precisely because financial data is comparatively scarce relative to the number of independent observations a nonparametric method needs to reliably outperform a good parametric assumption. Nonparametric and semi-parametric methods (trees, kernel methods, nearest-neighbour approaches) earn their place specifically where a genuinely nonlinear, unknown-shape relationship is suspected and enough data exists to support the extra flexibility — high-frequency data, or pooled panels across many instruments, are common places this trade tips in their favor.