Determining the Number of Factors

Run principal component analysis on a covariance matrix of a thousand stocks and you get, technically, a thousand principal components, each one explaining a little more of the total variance than the last. Keep all of them and you have not built a risk model, you have just rewritten the original covariance matrix in a different basis, with all its estimation noise intact. Keep too few and you throw away real, systematic risk that the market actually shares. Somewhere between "one factor" and "a thousand factors" is the right number, and choosing it is a genuine statistical problem, not a matter of taste.

The analogy

A photograph can be compressed by keeping only its most important visual "components," throw away too many and the image turns to mush, missing real structure; keep too many, including the sensor's random pixel noise, and the file barely compresses at all, no more useful than the raw original. Somewhere in between is the point where you have kept everything that looks like a real edge or shape and discarded everything that looks like static. Choosing the number of statistical risk factors is exactly this same tradeoff, applied to a covariance matrix instead of a photograph.

Building the decision

Each principal component $k$ explains a share of total variance equal to its eigenvalue $\lambda_k$ divided by the sum of all eigenvalues. A crude, classic rule (the "scree plot") looks for where the sequence of eigenvalues, sorted largest to smallest, stops dropping sharply and flattens out into a long, gently-declining tail:

$\text{explained share}_k = \frac{\lambda_k}{\sum_{j=1}^{N}\lambda_j}.$

In words: rank every component by how much of the total wobble in the data it accounts for, and look for the point where adding one more component barely helps anymore, the "elbow" in the plot. A more rigorous approach, due to Bai and Ng, treats the choice itself as a statistical estimation problem: it penalizes adding more factors (the way Regularization as a Prior penalizes model complexity in regression) and picks the number $k$ that minimizes a combined score of unexplained variance plus a complexity penalty that grows with both the number of factors and the size of the dataset.

$IC(k) = \ln\left(\hat{\sigma}^2_k\right) + k \cdot g(N, T),$

where $\hat{\sigma}_k^2$ is the leftover unexplained variance with $k$ factors, and $g(N,T)$ is a penalty term that grows with the number of factors $k$, scaled by how large the panel is (number of stocks $N$ and time periods $T$). In plain words: adding factors always shrinks the first term, so the penalty exists purely to stop you from adding factors forever; the chosen $k$ is wherever the tradeoff is optimized.

Picture unexplained variance following a curve shaped like this in reverse, falling fast at first as the first few factors are added, then flattening: the informative structure lives in that steep early drop, and the flat tail is largely noise.

Worked example 1: a scree plot by hand

Ten stocks' covariance matrix yields eigenvalues (sorted) of $5.2, 2.1, 0.9, 0.4, 0.3, 0.25, 0.2, 0.18, 0.15, 0.12$, summing to $9.7$. Explained shares: $5.2/9.7 = 53.6\%$, $2.1/9.7 = 21.6\%$, $0.9/9.7 = 9.3\%$, then a long tail under 4.5% each. The first two components explain $75.2\%$ of total variance, and the drop from component 2 to 3 is sharper than any later drop, marking the elbow at $k=2$.

Worked example 2: comparing two factor counts on the same portfolio

A ten-stock equal-weighted portfolio's risk is estimated two ways: with $k=2$ factors, predicted volatility is $11.5\%$ annualized; with $k=5$, it's $12.8\%$. Backtesting both against 24 months of realized returns, actual volatility came in at $12.6\%$. The $k=5$ forecast is closer, evidence that stopping at the scree plot's elbow discarded real, priced risk a richer factor count captured.

What this means in practice

Commercial statistical risk models typically settle on somewhere between 10 and 50 factors for a broad equity universe, not because that number is theoretically sacred, but because it is where information-criterion-style tests and out-of-sample backtests, run repeatedly across many samples, tend to land for that market's typical noise level.