Temperature Scaling for Neural Networks
A one-parameter fix applied after training that rescales a neural network's overconfident output probabilities so they actually match how often predictions turn out to be correct.
Prerequisites: Probability Calibration
Modern neural networks are often very confident but wrong in a systematic way: a classifier might output 95% confidence for a prediction that is only actually correct about 70% of the time. Temperature scaling is a simple post-hoc fix — after training is finished, divide the network's raw pre-softmax scores (logits) by a single learned scalar, the temperature, before applying softmax again, and the resulting probabilities become much better calibrated.
Temperature scaling fixes overconfident probabilities by dividing the network's logits by one learned number before the final softmax, which softens (or sharpens) every prediction uniformly without changing which class the model picks.
Why it works without hurting accuracy
Dividing all logits by a temperature greater than 1 shrinks the gaps between them, which softens the resulting probabilities toward less extreme values, while dividing by a value less than 1 sharpens them. Crucially, this rescaling is monotonic — it never changes which class has the highest score — so the model's actual predictions (and its accuracy) are completely unchanged; only the reported confidence shifts. The single temperature value is fit on a held-out validation set by minimizing calibration error, not accuracy.
It is popular precisely because it is cheap: one number, fit in seconds after training is otherwise complete, with no retraining and no change to the model's actual predictions.
Related concepts
Practice in interviews
Further reading
- Guo, C. et al., 'On Calibration of Modern Neural Networks' (2017)