The Rayleigh Quotient

Given a symmetric matrix $A$ — a covariance matrix, say — and a candidate direction $x$, how much does $A$ stretch things along that direction? The Rayleigh quotient answers exactly this:

$$R(x) = \frac{x^\top A x}{x^\top x} .$$

In plain English: the numerator measures how much $A$ stretches $x$ (projected back onto $x$ itself), and dividing by $x^\top x$ normalizes away the length of $x$, so $R(x)$ only depends on $x$'s direction, not its size. If you scan over every possible direction $x$, the maximum value $R(x)$ ever reaches is exactly $A$'s largest eigenvalue, achieved when $x$ is the corresponding eigenvector; the minimum value is $A$'s smallest eigenvalue, achieved at that eigenvector. This turns an eigenvalue problem into an optimization problem — maximizing a ratio — which is precisely the trick behind algorithms like power iteration and behind PCA's search for the direction of maximum variance.

Worked example. For $A = \begin{pmatrix}2 & 0\\0 & 5\end{pmatrix}$, try $x = (1, 0)$: $R(x) = 2/1 = 2$. Try $x = (0,1)$: $R(x) = 5/1 = 5$. Try the diagonal direction $x=(1,1)$: $R(x) = (2 + 5)/2 = 3.5$, between the two. No direction ever beats $5$ or falls below $2$ — exactly $A$'s two eigenvalues.