The p Greater Than n Regime

Suppose someone asks you to draw the one true straight line through a single data point. You can't — a line has two degrees of freedom (a slope and an intercept), and one point isn't enough to pin both down; infinitely many different lines pass through it exactly. Give them a second point and, generically, exactly one line fits both. This isn't really about lines specifically — it's about having fewer independent facts than unknowns. When the number of candidate predictor variables $p$ exceeds the number of observations $n$, you're in exactly this situation, just scaled up: you're trying to determine $p$ regression coefficients from only $n$ rows of data, and when $p > n$ there are infinitely many combinations of coefficients that fit the data perfectly. This is the "p greater than n" regime, and it is a qualitatively different problem from ordinary small-sample noise — it's not that the answer is uncertain, it's that there is no unique answer to be uncertain about, without bringing in extra assumptions from outside the data.

Why it breaks, one symbol at a time

Ordinary least squares solves for coefficients $\hat\beta$ by minimizing squared error, and the closed-form solution requires inverting the matrix $X'X$, where $X$ is the $n \times p$ matrix of predictor values:

$$\hat\beta = (X'X)^{-1} X'y$$

In words: the fitted coefficients come from "undoing" the predictor-to-predictor relationships captured in $X'X$ and applying that to how the predictors relate to the target $y$. The matrix $X'X$ is $p \times p$, and its rank can never exceed $\min(n, p)$ — a basic fact about matrices built by multiplying an $n \times p$ matrix by its own transpose. When $p > n$, the rank of $X'X$ is at most $n$, strictly less than $p$, which means $X'X$ is singular — not invertible — and $(X'X)^{-1}$ simply does not exist. The regression doesn't have one right answer with some uncertainty around it; it has an entire subspace of solutions that all achieve exactly zero training error, and nothing in the data itself picks among them.

Worked example 1: an explicitly singular matrix

Take $p=3$ candidate features but only $n=2$ observations:

$$X = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 1 & 4 \end{pmatrix}$$

Then $X'X$ is $3\times3$:

$$X'X = \begin{pmatrix} 5 & 4 & 11 \\ 4 & 5 & 10 \\ 11 & 10 & 25 \end{pmatrix}$$

Its determinant works out to exactly $0$ (checkable by cofactor expansion, or by noting a $3\times3$ matrix built from only 2 independent rows of underlying data cannot have full rank 3) — confirming $X'X$ is singular, so $(X'X)^{-1}$ doesn't exist and OLS has no unique solution. Now contrast with $n=4 > p=3$, using an additional two observations $(0,1,1)$ and $(3,0,2)$ added to $X$: the resulting $4\times3$ matrix generically has rank $3$ (full column rank), and the corresponding $X'X$ has a nonzero determinant, so it's invertible and OLS returns exactly one answer.

Worked example 2: ridge regression restores a unique answer

Ridge regression adds a small positive constant $\lambda$ times the identity matrix before inverting:

$$\hat\beta_{\text{ridge}} = (X'X + \lambda I)^{-1} X'y$$

In words: nudge every diagonal entry of $X'X$ up by $\lambda$ before inverting — this guarantees invertibility regardless of rank, because adding $\lambda I$ to a matrix shifts every eigenvalue up by $\lambda$, and a matrix with all-positive eigenvalues is always invertible. Using the same $p=3$, $n=2$ example, take $\lambda = 1$: $X'X + I = \begin{pmatrix}6&4&11\\4&6&10\\11&10&26\end{pmatrix}$. This matrix now has a nonzero determinant (unlike the un-regularized version), so it can be inverted and $\hat\beta_{\text{ridge}}$ is a single, well-defined vector even though $p > n$. The cost is that $\hat\beta_{\text{ridge}}$ is now biased toward zero relative to whatever the "true" coefficients might be — you've traded the complete non-identifiability of raw OLS for a unique but shrunk, and therefore biased, answer. The alternative path is reducing $p$ first — via feature selection, principal components, or an explicit sparsity assumption (most true coefficients are exactly zero) — so that ordinary methods apply to a smaller, well-posed problem.

What this means in practice

• This is the default regime for cross-sectional equity research with a large factor library. Hundreds of engineered signals or alternative-data features tested against a modest number of independent time periods routinely puts $p$ comfortably above $n$ (in the relevant, effectively-independent sense), so lasso-type sparsity assumptions or heavy dimension reduction aren't optional extras — they're required for the estimation problem to be well-posed at all.
• High-dimensional risk models face the same wall when the number of securities or candidate factors exceeds the number of independent history days — this is precisely why factor covariance estimation leans on structure (a small number of common factors) rather than a raw high-dimensional covariance matrix.
• Cross-validation still "runs" in software even when $p > n$ without regularization — a pseudo-inverse will silently return some numbers — which is exactly why this failure mode is dangerous rather than merely inconvenient; nothing crashes, but the output has no statistical meaning as an estimate of anything.