Factor Risk Models

A 3,000-stock covariance matrix has over four million distinct entries, vastly more than any sample can estimate (Covariance Matrix Estimation). Factor risk models resolve this by imposing structure: asset returns are driven by a handful of common factors plus stock-specific noise. This is both an economic statement (stocks co-move because they share exposures to the market, sectors, value, size, momentum) and a statistical device that shrinks the number of parameters from $O(N^2)$ to $O(NK)$, making the covariance matrix estimable, invertible, and interpretable. It is the backbone of every institutional risk system (Barra/MSCI, Axioma, Northfield) and the language of performance attribution.

The factor structure

Model each asset's excess return as a linear combination of $K$ common factor returns $f_t$ plus an idiosyncratic residual:

$r_{it} = \sum_{k=1}^{K} B_{ik} f_{kt} + \varepsilon_{it}, \qquad \text{or in matrix form}\quad r_t = B f_t + \varepsilon_t.$

Here $B$ is the $N\times K$ matrix of factor loadings (exposures), $f_t$ is the $K$-vector of factor returns, and $\varepsilon_t$ is specific (idiosyncratic) return. The two identifying assumptions: idiosyncratic returns are uncorrelated across assets ($\operatorname{Cov}(\varepsilon_i,\varepsilon_j)=0$ for $i\ne j$) and uncorrelated with the factors. The Capital Asset Pricing Model (CAPM) is the one-factor special case ($K=1$, factor = market, $B$ = beta).

The covariance decomposition

Under those assumptions the asset covariance matrix factorizes:

$\boxed{\;\Sigma = B\,F\,B^\top + D,\;}$

where $F = \operatorname{Cov}(f)$ is the $K\times K$ factor covariance matrix and $D = \operatorname{diag}(\sigma_{\varepsilon_1}^2,\dots,\sigma_{\varepsilon_N}^2)$ is the diagonal matrix of specific variances. The first term $BFB^\top$ is systematic (common, undiversifiable) risk; $D$ is idiosyncratic risk that diversifies away in a large portfolio. Count the parameters: $B$ has $NK$, $F$ has $K(K+1)/2$, and $D$ has $N$, total $O(NK)$ versus $O(N^2)$ for the full matrix. For $N=3000$, $K=50$: about $150{,}000$ parameters instead of $4.5$ million. That is why the factor covariance is well-conditioned and invertible where the sample matrix is not.

Three flavors of factor model

1. Fundamental (Barra-style). Loadings $B$ are observable firm characteristics, industry membership, size (log market cap), book-to-price, momentum, leverage, volatility. Each period, cross-sectionally regress returns on these known exposures to estimate the factor returns $f_t$. Robust and interpretable; the industry default.
2. Macroeconomic (Chen-Roll-Ross style). Factors are observable macro series, industrial production, inflation surprises, term spread, credit spread. Loadings $B$ are estimated by time-series regression of returns on the factors.
3. Statistical (PCA-based). Both $B$ and $f$ are latent, extracted from the return covariance by PCA (Principal Component Analysis)/factor analysis. Maximally explanatory in-sample but the factors lack economic labels and can be unstable.

Estimating the pieces

For a fundamental model: at each date run a cross-sectional Ordinary Least Squares (OLS) (usually GLS, weighted by inverse specific variance) of that day's returns on the exposure matrix $B$ to get the factor return $f_t$. Accumulate a history of $\{f_t\}$ and estimate $F$ from it (with shrinkage/EWMA). Specific variances $D$ come from the residual returns $\hat\varepsilon_{it}$, often shrunk toward a cross-sectional or industry average because single-stock residual vols are noisy. The full risk model $\Sigma = BFB^\top + D$ then delivers portfolio risk $\sigma_p = \sqrt{w^\top\Sigma w}$.

Risk attribution

The real payoff is decomposition. A portfolio's factor exposures are $x = B^\top w$ (a $K$-vector), and its variance splits cleanly:

$\sigma_p^2 = \underbrace{x^\top F x}_{\text{factor (common) risk}} + \underbrace{w^\top D\, w}_{\text{specific risk}}.$

You can attribute risk to each factor (how much of my vol is momentum exposure? sector concentration?), separate the intended bets from unintended ones, and compute a position's marginal contribution to tracking error. This is what makes factor models indispensable for portfolio construction and for Value at Risk (VaR) on large books.

Worked example

A single-factor (market) model with $\beta_i = 1.2$, market vol $\sigma_M = 16\%$, and specific vol $\sigma_\varepsilon = 25\%$. Total asset variance is $\sigma_i^2 = \beta_i^2\sigma_M^2 + \sigma_\varepsilon^2 = 1.2^2(0.16)^2 + (0.25)^2 = 0.0369 + 0.0625 = 0.0994$, so $\sigma_i = 31.5\%$. Systematic risk is $\sqrt{0.0369} = 19.2\%$, specific $25\%$. Now hold an equal-weighted portfolio of 100 such (independent-residual) stocks each with $\beta=1.2$: the factor part stays at $19.2\%$ (undiversifiable, every stock shares it), but the specific part shrinks by $\sqrt{100}$ to $25\%/10 = 2.5\%$. Portfolio vol $\approx \sqrt{0.0369 + 0.000625} = 19.4\%$, almost pure factor risk. This is the CAPM lesson made mechanical: diversification kills idiosyncratic risk and leaves systematic risk.

Failure modes

• Missing factors. If a real common driver is omitted, its variance leaks into $D$, the residuals become correlated (violating the diagonality assumption), and portfolio risk is understated, dangerous for a book built to be "market-neutral" on the modeled factors.
• Unstable loadings. Characteristic exposures and betas drift; stale $B$ misprices current risk.
• Correlation spikes / crowding. In stress, idiosyncratic residuals co-move (the 2007 quant quake), the "diagonal $D$" fiction breaks, and diversification fails exactly when needed.
• Factor covariance estimation. $F$ itself needs shrinkage/EWMA; a badly estimated factor covariance propagates into every position.

In interviews

Write $r = Bf + \varepsilon$ and immediately give the payoff $\Sigma = BFB^\top + D$, then make the parameter-counting argument: $O(NK)$ instead of $O(N^2)$, which is why the factor covariance is estimable and invertible when the sample matrix is not. Distinguish systematic ($BFB^\top$) from specific ($D$) risk and explain the diagonal-$D$ assumption and what breaks when a factor is missing (correlated residuals). Name the three model types, fundamental/Barra, macro, statistical/PCA, and be ready to attribute portfolio variance to factor vs specific components. Connect it back: CAPM is the one-factor case. See Covariance Matrix Estimation and PCA (Principal Component Analysis).