Estimating Live Performance Without Labels

A model's true labels — whether a trade was actually profitable, whether a 30-day default happened, whether a forecast was realized — often aren't known for days, weeks, or months after a prediction is made. Waiting for those labels to arrive before checking whether the model is still working means finding out about a problem long after it started. Performance estimation without labels is the set of techniques for getting an earlier, imperfect read using only the information available immediately: the inputs and the model's own predictions.

Two workable proxies

Confidence-based estimation uses the model's own predicted probabilities as a stand-in for accuracy: if a classifier starts producing many more predictions clustered near 0.5 (maximum uncertainty) instead of its usual confident 0.1-or-0.9 pattern, that shift itself is a signal something has changed, even without knowing whether any individual prediction was right. Distribution-matching estimation compares the current input distribution against the training distribution (much like a population stability index check) and reweights or extrapolates the model's known training-time accuracy on similar-looking historical segments to estimate what accuracy it's likely getting now — the logic being that if today's inputs look statistically like a historical period where the model scored 85% accuracy, today's accuracy is probably in that neighborhood too, absent evidence otherwise.

Neither method replaces eventually checking against real labels — they're a bridge to catch large, obvious degradations early, not a permanent substitute for ground truth.

Worked example

A default-prediction model's true labels (whether a loan actually defaulted) aren't known for 90 days. To get an earlier signal, the team tracks the model's predicted-probability distribution daily. During the training period, predictions were bimodal — clustered near 0.05 (low risk) or 0.6 (high risk) — with an average predicted default rate of 4.2%. Three weeks after a new cohort of loans starts scoring, the average predicted probability climbs to 7.8% and the distribution shifts toward the middle, with far more borderline 0.3–0.4 predictions than before. No labels are available yet to confirm actual default rates, but the shift in the model's own confidence pattern — combined with a rising population stability index on the input features — is treated as an early warning, and the desk tightens underwriting thresholds provisionally while waiting for the 90-day labels to confirm or refute the concern. When labels do arrive, the realized default rate for that cohort comes in at 7.1%, close enough to the early estimate to validate the approach.

What this means in practice

Any model where true labels lag by more than a few days needs a proxy monitoring layer, or problems are discovered on a delay proportional to that label lag — for a 90-day default model, that's three months of continued bad decisions before the first direct evidence arrives. Proxies should be validated retrospectively against real labels whenever they do arrive, to keep calibrating how much to trust the early warning going forward.