Low-Rank Approximation and the Eckart-Young Theorem

A large covariance matrix of stock returns is genuinely high-dimensional, but most of its structure is driven by a handful of dominant factors, a market factor, a couple of sector factors, with the rest being noise. If you want to compress that matrix down to its "k most important patterns" and discard the rest, you need to know: what does the best possible rank-$k$ approximation even look like, and how do you know nothing else does better? The Eckart-Young theorem answers both questions with a strikingly simple construction.

An analogy: a photo compressed to its main brushstrokes

Think of a matrix as a grayscale image, a grid of pixel intensities. Compressing that image to look as close to the original as possible using only a handful of "brushstrokes" (simple rank-one patterns, each a single horizontal streak times a single vertical streak) is exactly the low-rank approximation problem. Eckart-Young says: the best set of brushstrokes to keep, for any target number $k$, are precisely the $k$ strongest ones the image naturally decomposes into, and adding any other combination of strokes, however cleverly chosen, can never beat keeping the top $k$ by strength.

The math, one symbol at a time

Every matrix $M$ has a singular value decomposition $M = U\Sigma V^\top$, where $\Sigma$ is diagonal holding the singular values $\sigma_1 \geq \sigma_2 \geq \cdots \geq 0$ in decreasing order, and $U$, $V$ hold the corresponding left and right singular vectors. The rank-$k$ truncated SVD keeps only the top $k$ singular values and their vectors:

The Eckart-Young theorem states that among all matrices of rank $k$ or less, $M_k$ is the one closest to $M$ (measured by summed squared entrywise error), and the size of the unavoidable error is exactly $\sum_{i=k+1}^{n} \sigma_i^2$, the total "strength" left in the singular values you discarded. In words: truncating the SVD isn't just a reasonable way to compress a matrix, it's provably the best one possible, and the leftover singular values tell you precisely how much accuracy you gave up.

Worked example 1: compressing a small matrix by hand

Let $M = \begin{pmatrix}3&0\\0&1\end{pmatrix}$, already diagonal so its singular values are $\sigma_1=3, \sigma_2=1$ with singular vectors along the axes. The best rank-1 approximation keeps only the top term: $M_1 = 3\begin{pmatrix}1\\0\end{pmatrix}\begin{pmatrix}1&0\end{pmatrix} = \begin{pmatrix}3&0\\0&0\end{pmatrix}$. The approximation error, entrywise squared, is $(0-0)^2 + (0-1)^2 = 1$, exactly $\sigma_2^2 = 1^2$, matching the theorem's prediction precisely, and no other rank-1 matrix can get closer to $M$ than this.

Worked example 2: factor-model compression of a return matrix

A covariance matrix for 100 stocks has singular values $\sigma_1 = 40$ (market factor), $\sigma_2 = 12$, $\sigma_3 = 8$ (sector factors), and 97 more small values summing in squares to about 50. Keeping a rank-3 approximation captures $40^2+12^2+8^2 = 1600+144+64=1808$ out of a total sum-of-squares of $1808+50=1858$, about $\frac{1808}{1858} \approx 97.3\%$ of the matrix's total "energy," using just 3 factors instead of 100. This is the mathematical justification for reducing a huge estimated covariance matrix to a handful of statistical factors: the discarded 96.7% of singular values contribute only 2.7% of total structure, most of which is likely estimation noise anyway.

Reducing a matrix's rank shrinks the ellipse the unit circle maps to down toward a line or point along its weakest singular directions, drag the entries to see which directions carry the most "stretch," the ones Eckart-Young says to keep.

What this means in practice

Eckart-Young is the theoretical backbone of factor models (compressing a covariance matrix to a few statistical factors), image and signal compression, and recommender systems (low-rank approximations of a ratings matrix). Whenever a quant reduces a large, noisy matrix to its "top $k$ components" via PCA or SVD truncation, they're implicitly invoking this theorem to know that choice is provably optimal, not just convenient.