Ill-Conditioning and the Hessian Spectrum

Gradient descent takes one step size and applies it in every direction at once, which works fine if the loss surface curves about the same amount in every direction — like the bottom of a round bowl. It works badly if the surface is shaped like a narrow ravine, curving steeply on the sides and gently along the floor, because the single step size that is safe for the steep direction is far too small to make real progress along the gentle one. That mismatch — ill-conditioning — is measured directly by the Hessian's eigenvalues.

The analogy: a round bowl versus a narrow ravine

Rolling a ball down a round, symmetric bowl, it heads straight for the bottom no matter which side it starts on. Rolling the same ball down a long, narrow ravine — steep walls, a nearly flat floor — it instead bounces back and forth between the walls, only slowly drifting along the floor toward the actual low point, because most of its motion gets absorbed zig-zagging across the narrow direction rather than progressing along the long one. Gradient descent on an ill-conditioned loss surface behaves exactly like that ball in the ravine: it oscillates across the steep-curvature directions while crawling along the gentle ones.

The condition number

The Hessian's eigenvalues describe curvature along each principal direction of the loss surface. The condition number is their ratio:

$$\kappa = \frac{\lambda_{\max}}{\lambda_{\min}}$$

where $\lambda_{\max}$ and $\lambda_{\min}$ are the largest and smallest eigenvalues. In words: $\kappa$ measures how lopsided the bowl is — $\kappa=1$ means perfectly round (equally steep in every direction), while $\kappa \gg 1$ means one direction is far steeper than another. Plain gradient descent's usable step size is capped by the steepest direction (a step too large there causes overshoot and divergence), roughly $\eta < 2/\lambda_{\max}$, even though that same step size is needlessly tiny for making progress along the gentlest direction — the larger $\kappa$ is, the worse this mismatch, and the more iterations are needed to converge.

Worked example 1: the safe step size and its cost

Suppose a quadratic loss has Hessian eigenvalues $\lambda_{\max}=100$, $\lambda_{\min}=1$, giving $\kappa = 100$. The safe learning rate bound is $\eta < 2/100 = 0.02$. Using $\eta = 0.05$ (bigger than the bound) along the steep direction causes the iterate to overshoot and oscillate with growing amplitude — diverging rather than converging. Using the safe $\eta=0.02$ instead, progress along the gentle direction (with curvature $\lambda_{\min}=1$) advances by roughly $\eta \times \lambda_{\min} = 0.02$ of the remaining distance per step — needing on the order of 100 steps just to meaningfully close that gap, purely because the step size had to be set small enough for the other direction.

Worked example 2: convergence speed scales with $\kappa$

For gradient descent on a quadratic, the number of iterations needed to shrink the error by a fixed factor scales roughly with $\kappa$. At $\kappa=1$ (perfectly conditioned), convergence takes on the order of a handful of iterations. At $\kappa=100$, it takes on the order of 100 times as many. At $\kappa=10{,}000$ — not unusual for a poorly-scaled neural network loss — it takes on the order of 10,000 times as many iterations to reach the same accuracy, all else equal. This single ratio, more than the raw size of the gradient, is what actually predicts how painfully slow plain gradient descent will be on a given loss surface.

Drag the matrix parameters here until the unit circle becomes a long, thin ellipse — that is exactly what a high condition number looks like geometrically: the eigenvector along the ellipse's long axis is the gentle direction, the short axis is the steep one.

What this means in practice

Ill-conditioning is a major reason adaptive optimizers (RMSProp, Adam, AdamW) and second-order methods exist — they effectively rescale each direction by its own local curvature, rather than using one global step size, directly attacking the mismatch a large condition number creates. Feature scaling and normalization layers also help indirectly, by keeping different parts of the network's loss surface from ending up with wildly different local curvatures in the first place.