Serving Models Inside a Trading System

Research code and production code answer different questions. A researcher's notebook asks: does this model predict well? A trading system asks a harder, less glamorous set of questions on top of that: can this model produce a prediction fast enough to be useful, can it be swapped for a new version without stopping the desk, and when it inevitably breaks — bad input, a dependency mismatch, a corrupted model file — does the system fail safely rather than trading on garbage. Serving is the layer that turns a model from a research artefact into something a live trading system can depend on, and most of what makes it hard has nothing to do with model accuracy.

The three problems serving actually has to solve

Latency. A model that takes 200 milliseconds to score is fine for a signal rebalanced daily and useless for a quoting decision that needs to happen in microseconds. The serving layer has to match inference speed to the decision cadence it feeds — which sometimes means a smaller, faster model deliberately traded off against a larger, more accurate one that can't meet the latency budget (see Inference Latency Budgets).

Version control and rollback. A model in production is a specific set of weights trained on a specific dataset at a specific point in time — not the code that produced it. If a new version is deployed and starts behaving badly, the system needs to identify exactly which version is live and revert to the last known-good one quickly, which requires treating trained model artefacts with the same discipline as a versioned software release, not as a file someone copied onto a server (see Model Rollback and Freeze Procedures).

Consistent, monitored inputs. A model in production sees whatever data arrives, including data that never appeared during training — a stock that just IPO'd with no history for a feature that needs 60 days of it, a data feed that goes stale, a value outside any range seen in training. Research code can quietly assume well-formed input; serving code has to explicitly decide what happens when that assumption breaks, because "the model produced a nonsense prediction because a feature was missing" is a silent failure that costs real money before anyone notices.

A worked example

A desk deploys a gradient-boosted signal model that scores every name in the universe once per minute during the trading day. In research, the model was validated on a clean, complete historical dataset where every feature was always present.

In production, roughly once a week, a data vendor's feed for one input feature goes briefly stale — the value simply stops updating for a few minutes rather than arriving as explicitly missing. The naive serving implementation forward-fills the last known value silently and keeps producing predictions, because nothing about a stale-but-present number looks obviously wrong to the code. The model, having never seen the "value frozen for several minutes" pattern in training, produces a scored signal that looks numerically plausible but is stale and uninformative — and because it doesn't fail loudly, the desk trades on it for several minutes before a separate data-quality monitor (unrelated to the model itself) flags the stale feed.

The fix is to make the serving layer explicitly track feature freshness — timestamp every input, not just its value — and refuse to produce a prediction (falling back to a neutral or previous-known-good signal, and alerting) whenever any required feature's timestamp is older than an explicit threshold, rather than trusting that a numerically valid-looking input is actually a fresh one.

Treat this curve as a stand-in for a health-check gate: below some threshold of input freshness or feature completeness, the system should sharply switch from "serve the model's prediction" to "fall back to safe behaviour," rather than degrading gradually and invisibly the way a raw prediction would.

Where serving breaks in practice

The most common gap is between the feature-computation code path used in research (often a batch job with the luxury of recomputing history) and the one used in live serving (built separately, under real-time constraints, sometimes by a different team). Even a subtle difference — a rounding convention, a different handling of a corporate action — means the model in production is being fed inputs that don't quite match what it learned from, which is a version of offline/online feature parity failure specific to the serving boundary.