Label Latency and Delayed Feedback

A model that predicts the next 20 trading days' return produces its guess in a fraction of a second. Finding out whether that guess was right takes 20 trading days, no matter how fast the model runs. This is label latency: the delay between a prediction being made and the true outcome being known. It sounds like a minor bookkeeping detail, but it is the reason a live trading system can be silently wrong for weeks with nobody able to prove it yet, and it drives real design decisions about how monitoring and retraining have to work.

Why the delay is unavoidable, and why it varies

Some labels arrive almost instantly: a click either happens or it doesn't, within seconds. Others take as long as the thing being predicted takes to happen: a 20-day return label is not known until 20 days pass; a prediction about whether a company defaults within a year is not known for up to a year; a prediction about the outcome of a merger arbitrage position is not settled until the deal closes or breaks, which can be months out. The label latency is not a system limitation to be engineered away — it is baked into what the model is being asked to predict. A shorter-horizon target has less latency, but only because it is answering a smaller, often less economically interesting question.

Worked example: how long before you can trust a live Sharpe estimate

A model predicts 20-day forward returns and gets one new prediction per stock per trading day. Suppose you want a live IC (information coefficient) estimate built from at least 60 resolved predictions before treating it as informative. Because each prediction needs 20 trading days to resolve, and predictions are generated daily, the first prediction resolves on day 21. To accumulate 60 resolved predictions you need 60 more trading days of predictions to have each resolved, so the 60th prediction (made on day 60) resolves on day 80. In other words, roughly 80 trading days — about four months — pass between the model going live and having enough resolved outcomes to say anything statistically meaningful about its live performance. For the first four months, anyone asking "is this working?" cannot get an answer from the labels alone.

What this means in practice

Because true labels lag so far behind live predictions, production monitoring leans heavily on proxies that don't require waiting: input drift (are the features still distributed like the training set?) and prediction drift (is the model's output distribution suddenly different — far more extreme scores, or a shift in which direction it favors?) can both be checked the same day. These are not substitutes for the real labels, but they are the only early-warning system available while the labels are still in flight. A second consequence is that any live retraining pipeline is necessarily working with a partial, censored sample: the most recent predictions in the training window have not resolved yet and cannot be included, which is a form of survivorship in time — only the labels that had time to resolve make it into the retraining set, systematically excluding the freshest and potentially most regime-relevant period.

Practice

1. A model predicts whether a company defaults within 12 months. Roughly how long after go-live can a firm compute its first reliable estimate of the model's precision on defaults, and why?
2. Why can input-drift monitoring detect a broken data pipeline within a day, while it can never on its own confirm that the model's predictions are still accurate?