Linear Independence and Rank

A risk model uses ten "factors" to explain a stock's return, but two of them — say, "value" and "cheap earnings yield" — turn out to move almost in lockstep. Feeding both into a regression doesn't add new information; it adds redundancy, and can make the regression's coefficients wildly unstable. Linear independence is the precise concept behind "does this vector add anything new," and rank counts how many genuinely independent directions a whole matrix contains.

An analogy: redundant witnesses

Imagine three witnesses describing a car accident. If two of them are actually the same person who gave a second, near-identical statement, you don't really have three independent pieces of evidence — you have two. Counting "how many witnesses" naively (three) overstates how much genuinely new information you have; the honest count treats the near-duplicate as adding almost nothing. Linear independence formalizes exactly this: does a new vector add a genuinely new direction, or can it already be built out of the others you have?

The mathematics, one symbol at a time

A set of vectors $v_1, \ldots, v_k$ is linearly independent if the only way to combine them to get the zero vector is by using all-zero coefficients:

$$c_1 v_1 + c_2 v_2 + \cdots + c_k v_k = 0 \implies c_1 = c_2 = \cdots = c_k = 0.$$

In words: no vector in the set can be built as a combination of the others — if it could, you could rearrange that combination to hit zero with a non-trivial mix of coefficients, so independence rules that possibility out entirely. If the vectors are the columns of a matrix $A$, the rank of $A$ is the largest number of columns (equivalently, rows) that form a linearly independent set — the true number of distinct directions the matrix's columns actually span, after accounting for any redundancy. A matrix is full rank if its rank equals the smaller of its number of rows and columns; falling short of full rank means at least one column (or row) is a combination of the others and carries no new information.

Worked example 1: checking independence by hand

Take $v_1 = (1, 2)$, $v_2 = (3, 1)$. Set $c_1(1,2) + c_2(3,1) = (0,0)$: $c_1 + 3c_2 = 0$ and $2c_1 + c_2 = 0$. From the first, $c_1 = -3c_2$; substituting into the second, $-6c_2 + c_2 = 0 \Rightarrow -5c_2 = 0 \Rightarrow c_2 = 0$, then $c_1 = 0$. Only the trivial solution works, so $v_1, v_2$ are independent — together they span the full 2-dimensional plane, rank 2. Now try $v_3 = (2, 4)$ alongside $v_1$: $v_3 = 2v_1$ exactly, so $c_1=2, c_2=-1$ gives $2v_1 - v_3 = 0$ with non-zero coefficients — dependent, adding no new direction.

Worked example 2: rank of a redundant factor matrix

A factor exposure matrix has three columns: momentum $m$, a "12-month momentum" column $m_{12}$ that happens to equal $1.02m$ (near-perfectly correlated in this toy example), and value $v$, independent of both. Because $m_{12} \approx 1.02 m$, the three columns don't span a 3-dimensional space — effectively only 2 independent directions exist (momentum-ish, and value), so the matrix has rank 2 despite having 3 columns. A regression using all three columns as separate regressors will struggle: it can't cleanly separate "momentum's effect" from "12-month momentum's effect" because there's really only one combined direction of information there to estimate a coefficient on.

Drag the transform above toward a degenerate (flattened) shape — when the two output vectors line up on the same line, the transform's columns have become linearly dependent, the matrix drops to rank 1, and the whole plane collapses onto a single line.

What this means in practice

Rank deficiency is the formal name for the multicollinearity problem in regression: near-dependent columns make a design matrix close to rank-deficient, inflating coefficient standard errors and making individual coefficients unreliable even when the model's overall fit looks fine. Rank also governs whether a matrix can be inverted at all (see Determinants and Matrix Inverses and Linear Systems) — a square matrix is invertible exactly when it is full rank.