Double Descent

The textbook story of model complexity is a single U-shape: test error falls as a model gets more flexible, then rises once it starts overfitting the training data. That story turns out to be incomplete. For many modern models — and even for some simple ones — pushing complexity past the point of perfectly fitting the training data, into a regime with far more parameters than data points, causes test error to fall a second time. This second dip is double descent, and it overturned the simple "just don't overfit" intuition that dominated machine learning for decades.

The analogy: a crowded highway that clears up further ahead

Traffic getting steadily worse as more cars merge onto a highway matches the usual overfitting story — more complexity, more congestion, worse outcome. But imagine that past a certain point, so many cars are on the road that a smart, coordinated traffic system kicks in and flow actually smooths out again, even more than before the congestion started. Double descent is the machine learning version: past the point where a model has just enough capacity to memorize the training data exactly (worst congestion), giving the model dramatically more capacity than that lets it find a smoother, better-generalizing fit than either the "just right" or "just barely overfit" models found.

Worked example: three models along the complexity axis

Consider fitting a model to 50 noisy training points with three levels of complexity:

• 10 parameters (underfit): training error moderate, test error moderate — the classic left side of the U.
• 50 parameters (exactly enough to fit every training point): training error near zero, but test error spikes — the model has one unique way to fit the data exactly, and that fit is jagged and noise-sensitive. This is the peak of the classic U, sitting exactly at the interpolation threshold.
• 5,000 parameters (wildly overparameterized): training error still near zero, but now there are countless ways to fit the training data exactly, and gradient-based training tends to land on a smooth, simple one among them (implicit regularization) — test error drops back down, often below where the 10-parameter model landed.

The peak sits precisely at the interpolation threshold — where the model has just enough capacity to fit the training data with (numerically) one essentially unique solution — because that solution is forced to chase every bit of noise with no freedom left to prefer a smoother alternative.

Push the complexity slider above past the usual overfitting point and notice the classic single-peak story it shows — that first rise is the same phenomenon as the double-descent peak here; double descent's twist is that pushing complexity even further can bring the error back down a second time, past what a plain bias-variance plot alone would suggest.

What this means in practice

Double descent is part of why very large neural networks — with far more parameters than training examples — routinely generalize well despite fitting their training data almost perfectly, something the classic bias-variance story would have predicted to be a bad sign. It doesn't mean "more parameters is always better" as a blanket rule; it means the danger zone to watch for is specifically the region right around the interpolation threshold, not complexity in general.