Recalibrating After Resampling
When training data is resampled to balance rare classes, a classifier's predicted probabilities become distorted and need to be recalibrated before they can be trusted as real probabilities.
Oversampling or undersampling the minority class is a common fix for class imbalance — it forces a model to actually learn the minority pattern instead of just predicting the majority class every time. But it comes with a cost: the model was trained on an artificial class ratio that no longer matches the real-world ratio, so its predicted probabilities come out systematically inflated for the minority class.
If the model was trained on data with an artificially even 50/50 split, but the real world has a 99/1 split, its predicted probability of "1" for a given case is too high relative to reality. Recalibration corrects this after the fact — usually by adjusting predicted probabilities with a known formula based on the resampling ratio, or by fitting a separate calibration model (like Platt scaling or isotonic regression) on a held-out, un-resampled validation set.
A model trained on resampled data can rank cases correctly but still gives probabilities that are systematically wrong — always recalibrate against the true class ratio before treating outputs as real probabilities.
Worked example
A fraud model is trained on data undersampled to a 50/50 fraud/non-fraud ratio, but real transactions are 99.5% legitimate. The model outputs a 0.80 fraud probability for a transaction. Without recalibration, a team might wrongly treat that as an 80% real-world chance of fraud. Applying the standard undersampling correction formula, adjusted for the true base rate, might reveal the real probability is closer to 3% — still flagged as high-risk relative to the tiny base rate, but nowhere near the raw 0.80 the model reported.
Practice in interviews
Further reading
- Dal Pozzolo et al., 'Calibrating Probability with Undersampling for Unbalanced Classification'