Implicit Regularization

Ridge regression explicitly tells the optimizer to prefer small weights by adding a penalty $\lambda \|w\|^2$ to the loss. Train an unregularized neural network with plain gradient descent — no penalty term anywhere in the loss — and it still, empirically, tends to find solutions that generalize reasonably well, not the wildly overfit solutions that classical theory would predict from an unconstrained, massively overparameterized model. The explanation is that the optimizer itself, not the loss function, is quietly doing the regularizing: among the infinitely many parameter settings that fit the training data equally well, gradient descent doesn't land on a random one — it systematically prefers a specific, structured one. That preference is implicit regularization.

The analogy: the shortest path a rolling ball takes

Imagine a valley with many points at the same lowest elevation — a flat trench, not a single lowest point. Drop a ball anywhere above the trench and let gravity and friction pull it down: it doesn't land at a random spot along the trench's floor, it rolls to the nearest point below where it started, following the path of steepest descent. Gradient descent on an overparameterized model is the same: among all the parameter settings achieving zero training loss, it converges to the one nearest its starting point, reached by the specific path steepest descent takes — not to an arbitrary interpolating solution. That "nearest along this specific path" selection is a form of preference, exercised without any explicit penalty term ever being written down.

The formal version for linear models

For linear regression with more features than data points ($p > n$), there are infinitely many weight vectors $w$ satisfying $Xw = y$ exactly. Starting gradient descent from $w_0 = 0$ and running it to convergence provably lands on

$$w_{\text{GD}} = \arg\min_{w} \|w\|_2 \quad \text{subject to} \quad Xw = y$$

Plain English: among every weight vector that fits the training data perfectly, gradient descent from zero converges to the one with the smallest Euclidean norm — exactly the same solution ridge regression converges to as $\lambda \to 0^+$. No penalty term was written into the loss; the minimum-norm bias came entirely from the optimizer's trajectory.

Worked example 1: two solutions, one path

Consider fitting $y = 4$ using a single feature $x=2$ with two parameters $w_1, w_2$ such that the model is $\hat y = (w_1 + w_2) x$. Any $w_1 + w_2 = 2$ satisfies the data exactly — e.g. $(2, 0)$, $(0, 2)$, or $(5, -3)$ all give zero training loss. Starting gradient descent from $(0,0)$ and running to convergence lands at $(1, 1)$ — the point on the line $w_1+w_2=2$ closest to the origin, i.e. minimum norm $\sqrt{1^2+1^2}=\sqrt2 \approx 1.41$, strictly smaller than $(2,0)$'s norm of $2$ or $(5,-3)$'s norm of $\sqrt{34}\approx 5.83$. Gradient descent didn't need to be told to prefer small weights; its symmetric path from the origin did it automatically.

Worked example 2: different initializations, different implicit bias

Repeat the same problem but start gradient descent from $(3, 0)$ instead of $(0,0)$. Convergence now lands at $(2, 0)$ — still on the line $w_1+w_2=2$, but the point closest to the new starting point $(3,0)$, not the origin. The implicit regularization is norm-minimizing relative to the initialization, not an absolute universal small-weight preference — which is exactly why weight initialization is not a cosmetic detail in deep learning: it changes which of the infinitely many zero-training-loss solutions the optimizer implicitly selects.

Watch how the descent path curves toward the loss minimum from wherever it starts — for an overparameterized model with a whole trench of minima, the specific point along that trench it lands on is determined by this path, which is the mechanism behind implicit regularization.

What this means in practice

Implicit regularization is a large part of why deep neural networks, trained with plain gradient-based methods and often no explicit weight penalty, still generalize despite having orders of magnitude more parameters than training examples — the optimizer's trajectory, its initialization, and stochastic noise from minibatches all contribute biases toward simpler, flatter, or lower-norm solutions. It also means two training runs that reach identical training loss are not interchangeable — different optimizers (SGD vs Adam), different initializations, or different batch sizes can implicitly regularize toward meaningfully different solutions with different test performance, even at identical training loss.