L-BFGS and Quasi-Newton Methods in ML

Plain gradient descent only knows which direction is downhill; it has no sense of how sharply the loss surface curves, so it often zig-zags down long narrow valleys. Newton's method fixes this by using the Hessian — the full matrix of second derivatives — to find the direction and step size that would reach the bottom of a locally quadratic bowl in one jump, but for a model with millions of parameters, storing and inverting that matrix is completely impractical. Quasi-Newton methods approximate the Hessian's effect instead of computing it directly, building up curvature information purely from the sequence of gradients seen so far. BFGS does this by updating a full approximate inverse-Hessian matrix after every step; L-BFGS ("limited-memory BFGS") keeps only the last handful (typically 5–20) of gradient and parameter changes and reconstructs an implicit curvature estimate from just those, so memory cost stays linear in the number of parameters instead of quadratic.

In practice this means L-BFGS can solve a smooth, deterministic optimization — for example fitting a logistic regression or calibrating a small pricing model on a fixed dataset — in a few dozen iterations where gradient descent might need thousands, because each step already accounts for how the slope itself is changing, not just its current value. It struggles, however, on the noisy, stochastic mini-batch gradients typical of deep learning, since its curvature estimate assumes each gradient measurement is trustworthy and consistent step to step; a gradient that jumps around from batch to batch corrupts the implicit Hessian approximation rather than refining it. That is why L-BFGS remains a default choice for calibration problems with a fixed, exact loss function — pricing model fits, small-scale statistical estimation — while Adam and plain SGD dominate large neural network training, where the loss itself is only ever estimated noisily from a sample.