Factor Analysis and Latent Variables

A student's scores across ten different subjects are all observed, but nobody can directly observe "general intelligence" — it is a latent variable, something assumed to exist and to drive the observed scores, but never measured on its own. Factor analysis is the statistical model built around exactly this setup: a small number of unobserved latent factors, plus test-specific noise that is unique to each observed variable, together generate everything you actually see. In finance, the observed variables might be dozens of stocks' returns, and the latent factor might be an unobserved "market regime" or "growth sentiment" driving all of them at once, with each stock also carrying its own private noise.

The model, in words before symbols

Factor analysis writes each observed variable as: a loading (a weight, how strongly this variable responds to the factor) times the latent factor, plus that variable's own private noise term. Crucially, the noise terms are assumed independent across variables — all of the shared co-movement between two stocks is assumed to flow entirely through the common factor, and everything left over is that stock's own business, uncorrelated with any other stock's leftover. This is a real, falsifiable assumption, and it is the thing that separates factor analysis from PCA.

$$x_i = \lambda_i f + \varepsilon_i$$

Read in words: the $i$-th observed variable $x_i$ equals its loading $\lambda_i$ times the shared latent factor $f$, plus its own private noise $\varepsilon_i$, where every $\varepsilon_i$ is independent of every other $\varepsilon_j$ and independent of $f$ itself.

Why this differs from PCA

PCA has no noise model at all — it treats every unit of variance in every observed variable as signal to be captured, and it decomposes the total covariance without distinguishing shared structure from private noise. Factor analysis explicitly splits variance into two buckets: the part driven by the shared factor (communality) and the part that is private noise (uniqueness), and it fits both. Concretely: suppose two stocks both have return variance of 4, and their observed correlation is 0.5, meaning their covariance is 2. A factor model with a single common factor could explain this by giving each stock a loading of $\lambda = \sqrt{2} \approx 1.41$ on the factor (so $\lambda_1 \lambda_2 = 2$ recovers the observed covariance), leaving each stock a private noise variance of $4 - 1.41^2 = 4 - 2 = 2$ — half of each stock's variance is "market," half is "idiosyncratic." PCA, run on the same two stocks, would instead report a single component that captures as much total variance as possible without ever asking how much of that variance was shared versus private.

Where PCA rotates axes purely to chase variance, factor analysis fits a specific generative story — factor plus independent per-variable noise — and asks which loadings make that story consistent with the observed covariances. Drag the transform above to see how a shared direction of stretch can still leave the two coordinates only partially aligned; that leftover misalignment is exactly the private noise factor analysis is built to separate out.

What this means in practice

Factor analysis underlies commercial risk models that decompose a stock's return into a small number of common factors (market, size, value, sector) plus stock-specific residual risk — the residual is treated as diversifiable, private noise precisely because the factor model assumes it is uncorrelated across stocks. That assumption is also the model's weak point: if two stocks share a real risk driver the model omits (say, both depend on the same supplier), their "private" noises will actually be correlated, and the model will understate the true covariance between them.