Principal Components Regression

A regression built on 20 different yield-curve maturities as predictors runs into trouble before it even starts: adjacent maturities — say, the 9-year and 10-year yields — move almost in lockstep, so the regression can't tell which one is "really" driving the outcome and which is just along for the ride. This is severe multicollinearity, and it makes the individual coefficients wildly unstable — small changes in the data can flip a coefficient's sign or magnitude, even though the model's overall predictions stay reasonable. Principal components regression sidesteps this by not regressing on the raw, entangled maturities at all: it first compresses the whole yield curve into a handful of uncorrelated summary variables that capture almost all the same information, then regresses on those instead.

An analogy: summarizing a chorus instead of transcribing every voice

Imagine trying to describe what a 20-person choir sounds like by writing down each singer's note individually — most of the singers are roughly harmonizing together, so their individual notes are highly redundant with each other, and small recording errors could make it look like two singers who were actually singing together were doing wildly different things. A far more stable and informative summary is to describe the chorus's overall pitch (are they mostly going up or down together), its harmonic spread (how much are voices splitting into different notes), and maybe one more dimension of variation — a handful of summary descriptions that capture almost everything meaningful about the whole chorus, without trying to pin down every individual, highly correlated voice. Principal components do exactly this to a set of correlated predictors: they find a small number of new variables — the dominant shared patterns of movement — that summarize almost all the variation in the original variables.

The mechanics, one symbol at a time

Given a set of predictors $X$, principal component analysis finds new variables $Z_1, Z_2, \dots$ — each a weighted combination of the original predictors, $Z_k = \sum_j w_{kj} x_j$ — chosen so that $Z_1$ captures as much of the variation across all the original predictors as possible, $Z_2$ captures as much of the remaining variation as possible while being uncorrelated with $Z_1$, and so on. Principal components regression then fits an ordinary regression using only the first few components:

$$y = \gamma_0 + \gamma_1 Z_1 + \gamma_2 Z_2 + \dots + \gamma_m Z_m + \varepsilon, \qquad m \ll p,$$

where $p$ is the original number of predictors and $m$ is a much smaller number of retained components. In plain English: instead of asking the regression to untangle $p$ correlated raw variables, you first do the untangling yourself via PCA — which is a purely geometric summary, computed without even looking at the outcome $y$ — and then regress on a small number of clean, uncorrelated summary variables, which is a numerically stable, well-behaved regression problem by construction, since the components are uncorrelated with each other.

Worked example 1: yield curve levels, slope, and curvature

A regression predicting a bond fund's monthly return from 15 individual Treasury yields (2-year through 30-year) suffers from severe multicollinearity — the correlation between adjacent maturities often exceeds 0.95. Running PCA on the 15 yields typically produces a first component that's almost exactly the average level of all yields (explaining, say, 85% of the total variation across maturities), a second component that resembles the slope — short rates versus long rates moving in opposite directions (explaining another 10%), and a third resembling curvature — the middle of the curve moving differently from both ends (explaining 3%). Together, these three components already capture 98% of the variation across all 15 original yields. Regressing the fund's return on just these three components — level, slope, curvature — instead of all 15 raw yields gives stable, interpretable coefficients, because the three components are uncorrelated with each other by construction.

Worked example 2: comparing prediction accuracy to the raw regression

On 10 years of monthly data, the raw 15-variable regression achieves an in-sample R-squared of 34%, but out-of-sample (on a held-out later period) it collapses to 8% — the unstable, wildly-varying coefficients from multicollinearity were badly overfitting sample-specific noise. Principal components regression using the first 3 components achieves a slightly lower in-sample R-squared of 31%, but an out-of-sample R-squared of 27% — far closer to its in-sample number, because the coefficients on the three stable, uncorrelated components don't swing wildly from one sample to the next the way the 15 raw, entangled coefficients did. The small in-sample cost of compressing to 3 components buys a large improvement in how well the model generalizes.

Watch how a stretch transformation has natural "axes" it stretches along most and least — those axes are exactly what PCA finds in a cloud of correlated predictors: the direction of greatest shared variation becomes the first principal component, the next-most-variation direction (at a right angle to the first) becomes the second, and so on.

What this means in practice

Principal components regression is a standard tool wherever predictors are numerous and structurally correlated by nature — yield curve points, a panel of similar valuation ratios, or overlapping technical indicators built from the same underlying price series. It trades a small amount of interpretability (a coefficient on "the second principal component" is less intuitive than a coefficient on "the 10-year yield") for much greater numerical stability and often meaningfully better out-of-sample performance than regressing on the raw, entangled variables directly. It's closely related to ridge regression (see Ridge and LASSO Regularization), which achieves a similar stabilizing effect by shrinking coefficients rather than discarding low-variance components outright.