Loss Landscape Geometry and Sharpness

Two networks trained on the same data can reach an identical training loss — say, both perfectly fit the training set — and yet one generalises well to new data while the other falls apart on it. The training loss number alone cannot tell you which is which. Something about where in the space of possible weights each network landed matters beyond how low the loss got there, and that something is the shape of the ground around that point.

The analogy: parking on a hillside versus parking in a bowl

Picture the loss as elevation over a landscape whose coordinates are the millions of weights in a network. Training is rolling a ball downhill until it stops. Two spots can sit at exactly the same low elevation: one is the bottom of a wide, gently sloped bowl, the other is a narrow crack at the bottom of a steep-walled canyon. Nudge the ball slightly in the bowl and it barely moves up in elevation. Nudge it the same amount in the canyon and it shoots up the wall. Real-world data is a small nudge away from the training data — a slightly different sample, slightly different noise — so a model resting in the bowl tolerates that nudge gracefully, while a model wedged in the canyon's sharp bottom sees its loss spike the moment reality deviates even slightly from what it memorised.

Measuring sharpness

For weights $\theta^*$ at a minimum of the training loss $L(\theta)$, curvature is captured by the Hessian, the matrix of second derivatives $H = \nabla^2 L(\theta^*)$. A common one-number summary of sharpness is the largest eigenvalue of that matrix:

$$\lambda_{\max}(H) = \max_{\|v\|=1} v^\top H v$$

In words: pick the single direction $v$ (out of every possible direction you could nudge the weights) in which the loss curves upward fastest, and measure how sharply it curves in that worst-case direction. A large $\lambda_{\max}$ means the steepest wall nearby is very steep — the canyon. A small $\lambda_{\max}$ means every nearby direction is gentle — the bowl. A cheaper, second-derivative-free proxy used in practice perturbs the weights and reads off the loss change directly:

$$\text{sharpness} \approx \max_{\|\epsilon\| \le \rho} \big[L(\theta^* + \epsilon) - L(\theta^*)\big]$$

In words: try the worst nearby perturbation of size up to $\rho$ and see how much the loss jumps. A flat minimum barely reacts; a sharp one jumps a lot for the same size nudge.

Worked example 1: two 1-D minima with the same loss

Approximate two minima locally as parabolas, both touching zero loss at their bottom. Minimum A: $L(\theta) = 0.5(\theta - 3)^2$. Minimum B: $L(\theta) = 20(\theta - 7)^2$. Both have $L = 0$ exactly at their centre — identical training loss.

Nudge each by $\epsilon = 0.2$ away from its centre. Minimum A: $L(3.2) = 0.5(0.2)^2 = 0.5(0.04) = 0.02$. Minimum B: $L(7.2) = 20(0.2)^2 = 20(0.04) = 0.80$ — forty times the loss increase for an identical-sized nudge. The curvature term ($0.5$ vs $20$) is exactly this problem's $\lambda_{\max}$: it is the second derivative of a parabola, and it directly multiplies how much a small shift in weights costs you. A test example that behaves like a small shift from the training data hurts model B forty times more.

Worked example 2: a two-model comparison with test degradation

Model A and Model B both hit training loss $0.05$ (measured as mean squared error over standardised returns). Their empirically measured sharpness (loss increase after a fixed-size random weight perturbation) is $0.01$ for A and $0.30$ for B.

On a held-out month, actual returns differ from the training distribution by roughly the perturbation size used in the sharpness test. Model A's test loss comes in at $0.05 + 0.01 = 0.06$ — barely worse. Model B's test loss comes in at $0.05 + 0.30 = 0.35$ — seven times worse than its own training loss, despite having tied Model A during training. If you only ever look at the training curve, the two models are indistinguishable until the moment the test set arrives and one of them breaks.

What this means in practice

Sharp minima are strongly associated with large-batch training: bigger batches give smoother, less noisy gradient estimates, which lets optimisation settle precisely into narrow crevices that noisier small-batch training would have bounced straight out of. That link is why large-batch training is notorious for a "generalization gap" — matching or beating small-batch training loss while doing measurably worse on held-out data — and why practitioners either shrink the learning rate less aggressively, warm up the learning rate, or explicitly optimise for flatness with methods like sharpness-aware minimization, which directly penalises the worst-case nearby loss rather than the loss at a single point. On financial data, where the live market is never exactly the training distribution, this matters more than on a static image benchmark: a model resting in a sharp minimum is a model one regime shift away from a large, sudden degradation, even though its backtest loss looked identical to a flatter, more robust alternative right up until deployment.

The practical takeaway is that model selection should never rely on training loss alone, and ideally not on a single held-out loss number either — an explicit sharpness probe, even a crude one like perturbing the trained weights slightly and re-measuring the loss, is a cheap way to distinguish a genuinely robust fit from one that merely got lucky on a particular validation split. Two models tied on both training and validation loss can still carry very different amounts of hidden fragility, and that fragility only becomes visible once the deployment data drifts even slightly from what both models were tuned against.

Push the step size up in the explorer and watch the trajectory either settle cleanly or bounce around near the bottom — that bouncing is closely related to why noisy, small-step optimisation tends to avoid the narrowest crevices in a real loss landscape, landing in flatter regions instead.