Model Rollback and Freeze Procedures

A newly retrained model gets deployed on schedule and, within an hour, starts producing predictions that look subtly off — not crashing, just behaving worse than the model it replaced. The right response isn't to debug the new model live while it keeps trading; it's to get back to the old, known-good model as fast as possible, and then debug at leisure. That "get back to known-good" capability has to be built in advance — it doesn't exist automatically just because a new model was deployed carefully.

The idea: keep the old version alive and ready, not deleted

A rollback reverts a production system to the previous model version, undoing a deployment. It only works cleanly if the previous version's complete artefact — weights, preprocessing, and metadata bundled together, per model packaging — was kept around rather than overwritten when the new model was deployed. If deploying $v_{t+1}$ overwrites the file that held $v_t$, there's nothing left to roll back to.

A freeze procedure goes further: during an especially sensitive period (right after a suspicious market event, or a known data-quality incident with a vendor), a team can deliberately hold production at its current version and refuse automatic redeployment, even if a newer model finished training and would normally be promoted. This buys time to investigate without a model version change happening underneath the investigation.

Both depend on the same discipline: every deployed model version stays kept, addressable, and reloadable — not just the most recent one.

Worked example: a rollback that works because of upstream discipline

A firm deploys model $v_{47}$ nightly, replacing $v_{46}$. Two hours in, monitoring shows $v_{47}$'s predictions have unusually high variance versus its own backtested behavior — a red flag distinct from ordinary market noise. The on-call engineer triggers a rollback: the system loads the stored artefact for $v_{46}$ (kept, not deleted) and performs the same atomic hot-reload swap used for ordinary deployments, just pointing at the previous file. Production is back on known-good $v_{46}$ within minutes, while the investigation into $v_{47}$ proceeds without live capital exposed to the suspect model.

Contrast a shop that only keeps the current artefact, deleting the prior one once a new deployment is confirmed. When $v_{47}$ misbehaves, there's no $v_{46}$ artefact to roll back to — the only options are retraining from a data snapshot (if one was kept) or accepting the bad model while a fix is built, both far slower and riskier than a rollback that could have taken minutes.

Read that curve loosely as "time to recover from a bad deployment" against "number of past model versions retained": recovery time drops sharply once even one prior version is kept on hand, and flattens out after that — the marginal benefit of keeping many versions is much smaller than the benefit of keeping at least the last one.

What this means in practice

Rollback capability isn't something to build after the first incident — by then it's too late for that one. It has to be standard deployment policy: every promoted version retained, a documented and tested rollback procedure using the same atomic swap as ordinary deployment, and clear ownership over who can trigger a freeze or rollback without sign-off delays during an active incident.