Saddle Points and Plateaus

Gradient descent stops moving whenever the gradient hits zero, and it is tempting to assume that means it found a minimum — the bottom of a bowl. But a zero gradient only means "flat in every direction I can currently see," and there are two very different kinds of flat spot: an actual minimum, where every direction slopes back up, and a saddle point, where some directions slope up and others slope down. In the high-dimensional loss surfaces neural networks are trained on, saddle points vastly outnumber true local minima, and the training dynamics near each look completely different.

The analogy: a mountain pass, not a valley floor

Standing at the bottom of a bowl-shaped valley, every direction you could step leads uphill — that is a true minimum. Standing at a mountain pass between two peaks, the ground is momentarily flat along the trail you're walking, but if you turned sideways, the ground slopes sharply down toward either peak's base. A saddle point is exactly this pass: flat (zero gradient) in the specific directions the optimizer happens to be moving through, while still sloping steeply in other directions — the flatness is real, but temporary and directional, not a sign that the search is over.

Reading curvature from the Hessian

At any point where the gradient is zero, the Hessian matrix (second derivatives) tells you which kind of flat spot it is. If every eigenvalue of the Hessian is positive, the point curves upward in every direction — a true local minimum. If every eigenvalue is negative, it's a local maximum. If eigenvalues have mixed signs — some positive, some negative — the point is a saddle: sloping up along the eigenvectors with positive eigenvalues, down along the ones with negative eigenvalues. As the number of parameters grows into the millions, the odds that all eigenvalues happen to share the same sign at a random flat point shrink dramatically, which is exactly why saddle points dominate high-dimensional loss surfaces while genuine bad local minima are comparatively rare.

Worked example 1: classifying a saddle by hand

Let $f(x,y) = x^2 - y^2$. The gradient is $(2x, -2y)$, which is exactly zero at the origin $(0,0)$. The Hessian is constant: $\begin{pmatrix}2 & 0\\0&-2\end{pmatrix}$, with eigenvalues $+2$ and $-2$ — one positive, one negative, so the origin is a saddle point, not a minimum. Checking directly: moving along the $x$-axis to $(0.5, 0)$ gives $f=0.25>0$ (uphill), while moving along the $y$-axis to $(0, 0.5)$ gives $f=-0.25<0$ (downhill) — exactly the mixed behavior the eigenvalue signs predicted, confirmed by simply plugging in numbers.

Worked example 2: how slowly a plateau is escaped

Consider $f(x) = x^4$ near $x=0$ — flat to a much higher order than a simple saddle, a genuine plateau. The gradient is $f'(x) = 4x^3$. At $x=0.1$, the gradient is $4(0.001)=0.004$ — tiny. With a learning rate of $0.1$, one gradient descent step moves $x$ to $0.1 - 0.1(0.004) = 0.0996$ — barely any progress at all. Even after 100 such steps the position barely creeps forward, because the gradient shrinks roughly as fast as the position approaches zero. This is why plateaus (not just saddles) can trap training for a very long time even without any negative-curvature direction to worry about — the optimizer isn't stuck, it's just moving at a crawl.

A saddle point's Hessian stretches space along one eigenvector and compresses it along the other, with opposite signs — drag here to see how a transform with mixed-sign behavior along its two eigenvector directions looks nothing like a simple uniform bowl.

What this means in practice

Training loss curves that plateau for long stretches before suddenly dropping again are usually not stuck at bad local minima — they are more often crawling across a saddle or a near-flat region, escaping once momentum or gradient noise pushes them along a negative-curvature direction. This is a large part of why momentum-based and adaptive optimizers, and simply not panicking during a flat stretch of the loss curve, matter as much as they do.