Recalibrating After Resampling

Oversampling or undersampling the minority class is a common fix for class imbalance, it forces a model to actually learn the minority pattern instead of just predicting the majority class every time. But it comes with a cost: the model was trained on an artificial class ratio that no longer matches the real-world ratio, so its predicted probabilities come out systematically inflated for the minority class.

If the model was trained on data with an artificially even 50/50 split, but the real world has a 99/1 split, its predicted probability of "1" for a given case is too high relative to reality. Recalibration corrects this after the fact, usually by adjusting predicted probabilities with a known formula based on the resampling ratio, or by fitting a separate calibration model (like Platt scaling or isotonic regression) on a held-out, un-resampled validation set.

Worked example

A fraud model is trained on data undersampled to a 50/50 fraud/non-fraud ratio, but real transactions are 99.5% legitimate. The model outputs a 0.80 fraud probability for a transaction. Without recalibration, a team might wrongly treat that as an 80% real-world chance of fraud. Applying the standard undersampling correction formula, adjusted for the true base rate, might reveal the real probability is closer to 3%, still flagged as high-risk relative to the tiny base rate, but nowhere near the raw 0.80 the model reported.