Low-Rank Approximation and the Eckart-Young Theorem

A large covariance matrix of stock returns is genuinely high-dimensional, but most of its structure is driven by a handful of dominant factors — a market factor, a couple of sector factors — with the rest being noise. If you want to compress that matrix down to its "k most important patterns" and discard the rest, you need to know: what does the best possible rank-$k$ approximation even look like, and how do you know nothing else does better? The Eckart-Young theorem answers both questions with a strikingly simple construction.

An analogy: a photo compressed to its main brushstrokes

Think of a matrix as a grayscale image, a grid of pixel intensities. Compressing that image to look as close to the original as possible using only a handful of "brushstrokes" (simple rank-one patterns, each a single horizontal streak times a single vertical streak) is exactly the low-rank approximation problem. Eckart-Young says: the best set of brushstrokes to keep, for any target number $k$, are precisely the $k$ strongest ones the image naturally decomposes into — and adding any other combination of strokes, however cleverly chosen, can never beat keeping the top $k$ by strength.

The math, one symbol at a time

Every matrix $M$ has a singular value decomposition $M = U\Sigma V^\top$, where $\Sigma$ is diagonal holding the singular values $\sigma_1 \geq \sigma_2 \geq \cdots \geq 0$ in decreasing order, and $U$, $V$ hold the corresponding left and right singular vectors. The rank-$k$ truncated SVD keeps only the top $k$ singular values and their vectors:

$$M_k = \sum_{i=1}^{k} \sigma_i \, u_i v_i^\top.$$

The Eckart-Young theorem states that among all matrices of rank $k$ or less, $M_k$ is the one closest to $M$ (measured by summed squared entrywise error), and the size of the unavoidable error is exactly $\sum_{i=k+1}^{n} \sigma_i^2$ — the total "strength" left in the singular values you discarded. In words: truncating the SVD isn't just a reasonable way to compress a matrix, it's provably the best one possible, and the leftover singular values tell you precisely how much accuracy you gave up.

Worked example 1: compressing a small matrix by hand

Let $M = \begin{pmatrix}3&0\\0&1\end{pmatrix}$, already diagonal so its singular values are $\sigma_1=3, \sigma_2=1$ with singular vectors along the axes. The best rank-1 approximation keeps only the top term: $M_1 = 3\begin{pmatrix}1\\0\end{pmatrix}\begin{pmatrix}1&0\end{pmatrix} = \begin{pmatrix}3&0\\0&0\end{pmatrix}$. The approximation error, entrywise squared, is $(0-0)^2 + (0-1)^2 = 1$ — exactly $\sigma_2^2 = 1^2$, matching the theorem's prediction precisely, and no other rank-1 matrix can get closer to $M$ than this.

Worked example 2: factor-model compression of a return matrix

A covariance matrix for 100 stocks has singular values $\sigma_1 = 40$ (market factor), $\sigma_2 = 12$, $\sigma_3 = 8$ (sector factors), and 97 more small values summing in squares to about 50. Keeping a rank-3 approximation captures $40^2+12^2+8^2 = 1600+144+64=1808$ out of a total sum-of-squares of $1808+50=1858$ — about $\frac{1808}{1858} \approx 97.3\%$ of the matrix's total "energy," using just 3 factors instead of 100. This is the mathematical justification for reducing a huge estimated covariance matrix to a handful of statistical factors: the discarded 96.7% of singular values contribute only 2.7% of total structure, most of which is likely estimation noise anyway.

Reducing a matrix's rank shrinks the ellipse the unit circle maps to down toward a line or point along its weakest singular directions — drag the entries to see which directions carry the most "stretch," the ones Eckart-Young says to keep.

What this means in practice

Eckart-Young is the theoretical backbone of factor models (compressing a covariance matrix to a few statistical factors), image and signal compression, and recommender systems (low-rank approximations of a ratings matrix). Whenever a quant reduces a large, noisy matrix to its "top $k$ components" via PCA or SVD truncation, they're implicitly invoking this theorem to know that choice is provably optimal, not just convenient.