Newton's Method for Optimization

Gradient descent tells you which direction is downhill, but not how far to step — take too small a step and it crawls, too large and it overshoots. It also treats every direction as equally curved, which is rarely true: a loss surface curves sharply in some directions and gently in others. Newton's method fixes both problems at once by using not just the slope but the curvature of the function to pick both the direction and the step length in one move.

An analogy: parking by feel vs parking by sight

Imagine rolling a ball toward the bottom of a valley in fog, feeling only the slope under your feet (gradient descent) — you take a step, feel the new slope, take another step, and it takes many small steps to reach the bottom. Now imagine someone hands you a topographic map showing not just the slope where you stand but exactly how the valley curves around you. You can now calculate, in one shot, exactly where the bottom of a bowl-shaped valley must be, and jump straight there. That map of curvature is the extra information Newton's method uses.

The method, one symbol at a time

For a function $f(x)$ that you want to minimize, let $\nabla f(x)$ be the gradient (the vector of slopes) and $H(x)$ be the Hessian (the matrix of second derivatives, describing curvature in every direction). Newton's method updates the current guess $x_k$ by

$$x_{k+1} = x_k - H(x_k)^{-1} \nabla f(x_k) .$$

In words: at your current point, fit a bowl-shaped (quadratic) approximation to the true function using its slope and curvature there, then jump directly to the bottom of that bowl. If the true function actually were a perfect bowl, this reaches the minimum in one step. Real functions aren't perfect bowls, so you repeat the process, each time re-fitting a bowl at the new point, until the steps become tiny.

The Hessian does the heavy lifting: in a direction where the function curves sharply (steep-walled), $H$ is large, so the step in that direction is automatically shrunk; in a direction where it's nearly flat, $H$ is small, so the step is stretched out. Gradient descent has no such correction and takes clumsy, equal-sized steps in every direction.

Worked example 1: a simple quadratic by hand

Minimize $f(x) = 3x^2 - 12x + 7$, starting from $x_0 = 0$. The gradient (first derivative) is $f'(x) = 6x - 12$, and the Hessian (second derivative) is $f''(x) = 6$. At $x_0 = 0$: $f'(0) = -12$. The update is

$$x_1 = 0 - \frac{-12}{6} = 2 .$$

Check: $f'(2) = 6(2) - 12 = 0$, exactly the minimum. Because $f$ is already an exact quadratic, Newton's method reaches the true minimum $x^\star = 2$ in a single step — plain gradient descent with a fixed step size would need many iterations to creep there.

Worked example 2: a curved (non-quadratic) function

Minimize $f(x) = x^4 - 4x^2$, starting from $x_0 = 3$. Here $f'(x) = 4x^3 - 8x$ and $f''(x) = 12x^2 - 8$. At $x_0 = 3$: $f'(3) = 4(27) - 24 = 84$, and $f''(3) = 12(9) - 8 = 100$. The update:

$$x_1 = 3 - \frac{84}{100} = 3 - 0.84 = 2.16 .$$

Repeating: at $x_1 = 2.16$, $f'(2.16) \approx 4(10.08) - 17.28 \approx 23.03$, $f''(2.16) \approx 12(4.67) - 8 \approx 48.0$, giving $x_2 \approx 2.16 - 0.48 \approx 1.68$. Continuing a few more iterations converges rapidly toward the true minimum near $x^\star = \sqrt{2} \approx 1.414$ — notice the step size shrinks fast once curvature is correctly accounted for, unlike a fixed-size gradient step.

Drag the starting point and compare a curvature-aware step against a plain gradient step: watch how the Newton-style jump lands far closer to the minimum in fewer moves when the surface is bowl-like, and notice what happens when the surface briefly curves the wrong way.

What this means in practice

Newton's method is the fast-converging workhorse behind many solvers for smooth, well-behaved problems — pricing model calibration, maximum-likelihood fitting, and implied-volatility root-finding all lean on it or its variants. Its speed comes at a cost: computing and inverting the Hessian is expensive for high-dimensional problems (a Hessian for $n$ parameters is an $n \times n$ matrix), which is exactly why quasi-Newton methods like BFGS exist — they approximate the curvature information cheaply instead of computing it exactly.