Residual Correlation and Missing Factors

A weather model that tracks temperature and rainfall but has never heard of wind will treat two coastal towns' unexplained daily surprises as independent — even though, on the days a storm system rolls through, both towns get hit by the same wind at once. The model isn't wrong about temperature or rainfall; it's blind to a real, shared force that couples the two towns' "leftover" noise together. A factor risk model has the same failure mode. It assumes that after subtracting out the factors it knows about, the remaining stock-specific noise — the residual — is independent across stocks. If a real, shared factor is missing from the model, that assumption is false, and stocks that share the hidden factor have residuals that move together, whether the model can see it or not.

Why the diagonal assumption breaks

A factor model estimates specific variance as a diagonal matrix $\mathbf{D}$ — no off-diagonal terms, meaning zero assumed correlation between any two stocks' residuals. The true residual covariance, if a factor is missing, looks more like:

$$\boldsymbol{\Sigma}_{\epsilon} = \mathbf{D} + \gamma_k^2\, \mathbf{X}_k \mathbf{X}_k^\top ,$$

In words: the true residual covariance $\boldsymbol{\Sigma}_\epsilon$ is the model's assumed diagonal $\mathbf{D}$ plus an extra term the model never sees — $\mathbf{X}_k$ is the vector of each stock's exposure to the missing factor $k$, $\gamma_k^2$ is that factor's variance, and $\mathbf{X}_k \mathbf{X}_k^\top$ is exactly the kind of correlated, off-diagonal structure a diagonal model assumes doesn't exist. Any two stocks that both load on the missing factor pick up a nonzero covariance term the risk report never adds in.

Drag the correlation slider down toward zero and watch the scatter loosen into a shapeless cloud — that's what a risk model assumes two stocks' residuals look like. Left near 0.6, the visible diagonal trend is what actually happens between two stocks that share a hidden factor the model never measured.

Worked example 1 — two stocks, one missing factor

Two mid-cap suppliers each have a modeled specific volatility of $8\%$, so the model's assumed residual covariance between them is $0$. In reality both depend heavily on a single large customer — a genuine shared risk the sector and style factors don't capture — giving their residuals a true correlation of $0.6$. For an equal-weighted $100 position split $50/$50 between them, the model's assumed specific variance is $0.5^2(8\%)^2 + 0.5^2(8\%)^2 + 0 = 32\%^2$, i.e. $\sqrt{32} \approx 5.66\%$. The true specific variance adds the missing covariance term: $32\%^2 + 2(0.5)(0.5)(0.6)(8\%)(8\%) = 32 + 19.2 = 51.2\%^2$, i.e. $\sqrt{51.2} \approx 7.16\%$ — the real specific risk is about 27% higher than the risk report shows, purely from the missing shared factor.

Worked example 2 — the illusion gets worse with more names

Extend the same idea to five similarly-exposed suppliers, each 20% weighted, each with the same $8\%$ specific volatility and pairwise residual correlation of $0.6$. The model, assuming independence, reports specific variance shrinking with diversification: $5 \times 0.2^2 \times 8\%^2 = 12.8\%^2$, i.e. $\sqrt{12.8} \approx 3.58\%$ — lower than either stock alone, exactly the diversification benefit a risk model is supposed to reward. The true variance adds all 10 correlated pairs: $12.8\%^2 + 10 \times (0.2)(0.2)(0.6)(8\%)(8\%) = 12.8 + 15.36 = 28.16\%^2$, i.e. $\sqrt{28.16} \approx 5.31\%$ — nearly 50% higher than the modeled figure. Adding more names that share the hidden factor made the reported risk look better while the true risk barely moved.

What this means in practice

Risk teams check for this by examining the actual correlation of realized residuals after fitting the factor model — if residuals that "should" be independent keep showing up correlated within an identifiable group (a supply chain, a shared customer, a regional exposure), that's the signature of a missing factor, and the fix is either adding that factor explicitly or, for statistical models, letting more estimated factors absorb the structure. See Finding The Exposure You Did Not Know You Had for how this is spotted on a live book.