Gradient Descent

Train a neural network with millions of parameters, calibrate a volatility surface to hundreds of option prices, or fit any model where there's no clean formula for the best answer — and almost always, underneath whatever fancy name the optimiser goes by, the actual mechanism is embarrassingly simple: look at which direction makes the error worse fastest, walk the opposite way by a small amount, and repeat. That's gradient descent, and its simplicity is deceptive — the entire practical difficulty of training modern models comes down to one question this basic idea leaves open: how big should each step be?

The analogy: descending a foggy mountain by feel

You're on a mountainside in thick fog, trying to reach the valley floor, with no map and visibility only a few feet. The only information available is what's under your own feet right now: which direction is steepest downhill from exactly where you're standing. A sensible strategy: face the steepest downhill direction, take one step, stop, feel around again, and repeat. Take steps too large and you might stride right over a narrow valley and end up climbing the opposite slope; take steps too small and you'll get down eventually, but agonisingly slowly. Gradient descent is this exact strategy, formalised: it never sees the whole mountain, only the local slope, and its entire performance hinges on step size.

Writing it down

For a function $f(x)$ you want to minimise, the gradient $\nabla f(x)$ points in the direction of steepest increase at $x$ (in one dimension it's just $f'(x)$; in many dimensions, it's the vector of all partial derivatives). Since you want to go down, you step in the opposite direction:

$$x_{k+1} = x_k - \eta\, \nabla f(x_k) .$$

In words: the next point $x_{k+1}$ is the current point $x_k$, moved a distance controlled by $\eta$ (the learning rate, or step size) directly opposite the direction of steepest increase — i.e. toward steepest decrease. Repeat this update over and over, and for well-behaved (convex, convex) functions with a small enough $\eta$, the sequence $x_k$ provably converges to the true minimum.

The choice of $\eta$ is where all the practical difficulty lives. Too large, and the update can overshoot the minimum and diverge — each step lands further from the answer than the last, oscillating out of control. Too small, and convergence is correct but glacially slow, needing enormous numbers of steps to make any real progress. For a simple case — a quadratic $f(x)=\tfrac12 a x^2$ — the exact stability boundary can be written down: the iteration converges if and only if $0 < \eta < 2/a$, and it converges fastest at $\eta = 1/a$, landing exactly on the minimum in a single step. This one clean formula is the seed of every learning-rate heuristic used in practice.

Watch a running quantity settle onto its true value as more steps accumulate — that settling curve is the same shape a well-tuned gradient descent run traces on its own error, step by step getting closer to the minimum, with the rate of settling controlled by the same trade-off between step size and stability.

Worked example 1: minimising a parabola, three steps by hand

Minimise $f(x) = x^2 - 4x + 5$ (true minimum at $x^\star=2$, found by $f'(x)=2x-4=0$, with $f(2)=1$). Gradient: $f'(x) = 2x-4$. Start at $x_0 = 0$, use learning rate $\eta = 0.2$.

Step 1: $f'(0) = -4$. Update: $x_1 = 0 - 0.2(-4) = 0.8$.

Step 2: $f'(0.8) = 2(0.8)-4 = -2.4$. Update: $x_2 = 0.8 - 0.2(-2.4) = 0.8+0.48=1.28$.

Step 3: $f'(1.28) = 2(1.28)-4=-1.44$. Update: $x_3 = 1.28-0.2(-1.44)=1.28+0.288=1.568$.

The sequence is $0,\, 0.8,\, 1.28,\, 1.568,\ldots$, converging toward $x^\star=2$. Notice the gap to the true answer — $2, 1.2, 0.72, 0.432$ — shrinks by a factor of exactly $0.6$ every step. That's not a coincidence: for a quadratic with $a=2$ and $\eta=0.2$, theory predicts the error shrinks by a factor of $|1-\eta a| = |1-0.4|=0.6$ per step, and the hand calculation matches exactly.

Worked example 2: overshoot, and choosing a better rate

Same function, same start $x_0=0$, but now $\eta = 1.2$ — larger than the stability boundary $2/a=2/2=1$ derived above, so divergence is predicted. Check it: $f'(0)=-4$, $x_1 = 0-1.2(-4)=4.8$. $f'(4.8)=2(4.8)-4=5.6$, $x_2 = 4.8-1.2(5.6)=4.8-6.72=-1.92$. $f'(-1.92)=2(-1.92)-4=-7.84$, $x_3=-1.92-1.2(-7.84)=-1.92+9.408=7.488$. The sequence is $0,\,4.8,\,-1.92,\,7.488,\ldots$ — swinging wider and further from $x^\star=2$ every step, exactly the overshoot-and-diverge failure mode predicted by $\eta > 2/a$.

Now try the theoretically optimal rate $\eta=1/a=0.5$: $f'(0)=-4$, $x_1 = 0-0.5(-4)=2$ — already exactly at the minimum, in a single step, matching the "fastest convergence" prediction exactly. In practice you rarely know $a$ (the curvature) in advance, which is precisely why real training pipelines tune the learning rate empirically or use adaptive methods that estimate curvature on the fly, rather than compute this ideal rate directly.

What this means in practice

• It's the literal engine of neural network training. Backpropagation computes $\nabla f$ efficiently for millions of parameters; gradient descent (almost always its noisy cousin, stochastic gradient descent) is what actually moves the parameters using that gradient.
• The learning rate is the single most consequential hyperparameter in most ML pipelines. Too high and training visibly diverges or oscillates (loss spikes); too low and training "works" but wastes enormous compute converging too slowly — this exact trade-off, proven analytically for a simple quadratic above, is what learning-rate schedules and warmup heuristics are built to manage on much messier, non-quadratic loss surfaces.
• It only guarantees a global minimum on convex problems. On the non-convex loss surfaces of deep networks, gradient descent still faithfully walks downhill, but "downhill" can lead to any of many local minima or saddle points — a very different guarantee from the quadratic case above.