Polynomial Regression and Basis Expansions

Linear regression can only draw a straight line, but a straight line is a poor fit whenever the true relationship curves — implied volatility against strike, say, which bends into a smile rather than staying flat. The fix doesn't require abandoning linear regression at all: create new features that are transformed copies of the original one — $x^2$, $x^3$, and so on — and hand all of them to the exact same linear regression machinery. The model is still, mathematically, a linear combination of features; it's just that some of those features happen to be nonlinear functions of the original input. That's a basis expansion, and polynomial regression is its simplest example.

The analogy: describing a curve with straight rulers of different lengths

Imagine trying to trace a curved coastline using only straight rulers. One ruler alone can't do it. But give yourself a ruler, a ruler-squared "bendable" tool, a ruler-cubed tool, and enough of them stacked with the right weights, you can trace almost any smooth curve — you're not bending any individual tool, you're combining straight, simple pieces in the right proportions. Basis expansion is exactly this: keep the combination step linear (just weighted addition), but make the ingredients being combined ($x$, $x^2$, $x^3$, $\sin x$, or anything else) as curved as needed.

The model

Ordinary polynomial regression of degree $d$ replaces a single feature $x$ with $d$ transformed copies:

$$\hat y = \beta_0 + \beta_1 x + \beta_2 x^2 + \dots + \beta_d x^d$$

Plain English: the prediction is a weighted sum of powers of $x$; the model curves because $x^2, x^3, \dots$ curve, but fitting it is still ordinary least squares — solve for $\beta_0, \dots, \beta_d$ exactly the way you'd solve for coefficients in plain linear regression, just on an expanded feature matrix that includes the powers as extra columns. More generally, a basis expansion replaces $x$ with any set of functions $\phi_1(x), \dots, \phi_m(x)$ — polynomials, sines and cosines, indicator functions for ranges of $x$ — and fits $\hat y = \sum_j \beta_j \phi_j(x)$ the same way.

Worked example 1: fitting a degree-2 curve by hand

Suppose three data points: $(x,y) = (1, 2), (2, 5), (3, 10)$, which look like $y = x^2 + 1$. Expand to features $[1, x, x^2]$: rows $[1,1,1], [1,2,4], [1,3,9]$. Solving the resulting $3\times3$ linear system exactly (three equations, three unknowns) gives $\beta_0=1, \beta_1=0, \beta_2=1$, i.e. $\hat y = 1 + 0\cdot x + 1\cdot x^2 = x^2+1$ — a perfect fit, because three points can always be fit exactly by a degree-2 polynomial (2 degrees of freedom plus intercept equals 3 parameters for 3 points), and the fitting procedure was ordinary least squares the entire time, just on expanded features.

Worked example 2: degree choice and overfitting on noisy data

Now take 10 noisy points scattered around a true relationship $y = 0.5x + \epsilon$ (a nearly straight line with small random noise). Fitting degree 1 (a line) gives a smooth fit with modest residuals, say a residual sum of squares (RSS) of 4.2 on held-out data. Fitting degree 9 on the same 10 training points drives training RSS to nearly 0 (a 9-parameter polynomial can pass through 10 points almost exactly) but held-out RSS explodes to something like 340 — the high-degree polynomial oscillates wildly between training points to hit each one exactly, a classic case of a basis expansion rich enough to memorize noise rather than capture the smooth underlying trend.

Adjust the curve's shape parameters here — the same mechanism (a fixed functional family, fit by adjusting coefficients) is what polynomial regression does, except the "coefficients" are chosen by least squares rather than by hand, and adding higher powers gives the fit more independent knobs to bend with.

What this means in practice

Basis expansions are the reason "linear regression" is far more flexible in practice than "fits a straight line" suggests — the same normal-equations machinery, the same interpretable coefficients, the same fast closed-form solve, applied to engineered nonlinear features. Quants use this constantly: fitting an implied-vol smile with a low-degree polynomial in log-moneyness, or capturing a nonlinear factor-return relationship by adding squared or interaction terms to an otherwise linear factor model, all while keeping the estimation, standard errors, and diagnostics of ordinary linear regression.