Label Smoothing

A classifier trained with an ordinary one-hot label — 100% probability on the correct class, 0% on every other — has an incentive to push its raw scores toward plus and minus infinity, because softmax only reaches an output of exactly 1 in the limit of an infinitely large logit gap. Chasing that unreachable target makes the model's weights grow large and its confidence badly calibrated: it becomes extremely sure of itself, including on inputs it's actually getting wrong. Label smoothing removes the impossible target by replacing it with an achievable one.

The analogy: never promising 100% certainty

A trader who says "I am 100% certain this trade wins" is either lying or hasn't thought about tail risk — a well-calibrated trader hedges even a high-conviction view, saying "90% likely, with 10% spread across everything that could go wrong." Label smoothing forces a model into the same habit at training time: instead of telling it "this example is the eagle class with 100% certainty, 0% everything else," it says "90% eagle, and split the remaining 10% evenly across every other class" — a target the model can actually reach without its weights running off to infinity.

The formula

For $K$ classes and smoothing parameter $\varepsilon$ (a small number like 0.1), the smoothed target for the true class $y$ becomes:

$$q_i = \begin{cases} 1 - \varepsilon + \dfrac{\varepsilon}{K} & i = y \\[4pt] \dfrac{\varepsilon}{K} & i \neq y \end{cases}$$

In words: instead of putting all probability mass on the true class, put $1-\varepsilon$ of it there, and spread the remaining $\varepsilon$ evenly across every class including the true one — so the true class ends up with a little more than $1-\varepsilon$, and every wrong class gets a small nonzero share instead of exactly zero.

Worked example 1: computing smoothed targets by hand

Suppose $K = 5$ classes and $\varepsilon = 0.1$. The true class gets $1 - 0.1 + 0.1/5 = 0.9 + 0.02 = 0.92$. Each of the other 4 classes gets $0.1/5 = 0.02$. Check it sums to 1: $0.92 + 4(0.02) = 0.92 + 0.08 = 1.0$. Compare to the unsmoothed one-hot target of $(1, 0, 0, 0, 0)$: the model is now being trained toward $(0.92, 0.02, 0.02, 0.02, 0.02)$ — still confidently pointing at the right answer, but never asked to reach an impossible logit gap.

Worked example 2: the effect on cross-entropy loss at convergence

Cross-entropy loss is $-\sum_i q_i \log p_i$ where $p_i$ is the model's predicted probability. With a one-hot target, loss is minimized only as $p_y \to 1$, requiring the pre-softmax logit for class $y$ to diverge to $+\infty$ relative to the others — loss keeps shrinking without ever hitting zero, so the optimizer keeps pushing weights larger indefinitely. With the smoothed target from example 1, loss is minimized when the model's own output probabilities match $(0.92, 0.02, 0.02, 0.02, 0.02)$ exactly — a finite, achievable set of logits, at which point the gradient is exactly zero and training stops pushing weights further.

What this means in practice

Label smoothing is a one-line change (label_smoothing=0.1 in most framework loss functions) and is standard in image classification and machine translation training recipes; it tends to improve calibration — how well a model's stated confidence matches its actual accuracy — which matters whenever a model's output probability is used as a sizing signal rather than just a top-1 prediction. It costs a small amount of peak training accuracy in exchange for that calibration and for some resistance to noisy or mislabeled training data.