The Rayleigh Quotient

Given a symmetric matrix $A$, a covariance matrix, say, and a candidate direction $x$, how much does $A$ stretch things along that direction? The Rayleigh quotient answers exactly this:

In plain English: the numerator measures how much $A$ stretches $x$ (projected back onto $x$ itself), and dividing by $x^\top x$ normalizes away the length of $x$, so $R(x)$ only depends on $x$'s direction, not its size. If you scan over every possible direction $x$, the maximum value $R(x)$ ever reaches is exactly $A$'s largest eigenvalue, achieved when $x$ is the corresponding eigenvector; the minimum value is $A$'s smallest eigenvalue, achieved at that eigenvector. This turns an eigenvalue problem into an optimization problem, maximizing a ratio, which is precisely the trick behind algorithms like power iteration and behind PCA's search for the direction of maximum variance.

Worked example. For $A = \begin{pmatrix}2 & 0\\0 & 5\end{pmatrix}$, try $x = (1, 0)$: $R(x) = 2/1 = 2$. Try $x = (0,1)$: $R(x) = 5/1 = 5$. Try the diagonal direction $x=(1,1)$: $R(x) = (2 + 5)/2 = 3.5$, between the two. No direction ever beats $5$ or falls below $2$, exactly $A$'s two eigenvalues.