Early Stopping Criteria and Patience

Training loss almost always keeps falling the longer you train — a model with enough capacity can eventually memorize the training set nearly perfectly. But loss on held-out data the model has never seen typically falls for a while, then starts rising again, as the model shifts from learning genuine patterns to memorizing training-set quirks. Training for a fixed, arbitrarily chosen number of epochs risks stopping too early, before the model has learned enough, or too late, after it has already started overfitting. Early stopping instead watches validation performance directly and stops training when that number stops improving.

The analogy: tasting soup while it simmers

A cook adjusting a soup by taste doesn't set a timer and walk away — they taste it periodically as it simmers, and stop the moment it goes from "still improving" to "getting worse" (over-reduced, over-salted). Stopping the instant the very first spoonful tastes slightly worse than the last would be jumpy — taste can wobble spoonful to spoonful even while the soup is still generally improving. So a sensible cook tolerates a few slightly-worse tastes in a row before concluding the soup has actually turned, rather than reacting to the very first blip. That tolerance for a few bad tastes in a row is exactly what "patience" means in early stopping.

The mechanics: monitor, track the best, wait out the noise

At the end of every epoch, evaluate the model on a held-out validation set and record the validation loss $L_{\text{val}}$. Track the best (lowest) validation loss seen so far, $L_{\text{best}}$, and a counter of how many epochs in a row have failed to beat it:

$$\text{if } L_{\text{val}} < L_{\text{best}}: \ L_{\text{best}} \leftarrow L_{\text{val}}, \ \text{counter} \leftarrow 0 \qquad \text{else}: \ \text{counter} \leftarrow \text{counter}+1$$

In words: whenever validation loss sets a new best, save that epoch's weights and reset the "epochs since improvement" counter to zero; whenever it fails to improve, increment the counter. Training stops once the counter reaches a chosen threshold called patience — the number of consecutive non-improving epochs tolerated before concluding the model has genuinely stopped improving rather than just having one noisy epoch. The final model used is not the one from the last epoch trained, but the one saved at $L_{\text{best}}$.

Worked example 1: patience of 3 stops training

Validation loss across epochs: $2.0, 1.8, 1.7, 1.75, 1.72, 1.80, 1.85$, with patience $=3$. Epoch 1: loss $2.0$, new best, counter $=0$. Epoch 2: $1.8 < 2.0$, new best, counter $=0$. Epoch 3: $1.7 < 1.8$, new best, counter $=0$. Epoch 4: $1.75 > 1.7$, no improvement, counter $=1$. Epoch 5: $1.72 > 1.7$, still no improvement (doesn't beat the best, $1.7$), counter $=2$. Epoch 6: $1.80$, counter $=3$ — patience threshold hit, training stops. The model saved and returned is the one from epoch 3, where validation loss $1.7$ was the best ever recorded, not the epoch 6 model that triggered the stop.

Worked example 2: patience too low reacts to noise

Same validation sequence, but patience $=1$. Epoch 4's non-improvement ($1.75 > 1.7$) immediately hits the patience threshold, and training stops right there — missing epoch 5's $1.72$, which, while not a new best, would have shown the model was still hovering near its best rather than clearly worsening. A patience this low can stop training prematurely on a single noisy epoch, before the model has had a fair chance to continue improving after a brief wobble.

What this means in practice

Early stopping is one of the cheapest and most broadly used forms of regularization — it costs almost nothing beyond periodic validation evaluation and requires no change to the model or loss function. Patience is itself a hyperparameter to tune: too low, and training stops on noise before real improvement resumes; too high, and you waste substantial compute training well past the point where the model was already at its best, though that waste is usually far cheaper than the alternative of overfitting badly.

Practice

1. Validation losses are $3.0, 2.5, 2.6, 2.4, 2.6, 2.7$ with patience $=2$. At which epoch does training stop, and which epoch's weights are kept?
2. Why is a patience of exactly 0 (stop the instant validation loss fails to improve) usually a bad idea?
3. Name one situation where a very noisy validation metric would argue for a higher patience value.