Model Serving Architectures

Training a model is the part everyone talks about. Getting its predictions into a live trading or risk system, fast enough and reliably enough to be useful, is a separate engineering problem with its own failure modes. A model that scores 0.05 information coefficient in research is worthless if the number reaches the trading engine 40 minutes after the market moved on the information it was based on. Serving architecture is the decision of when and how a prediction gets computed and delivered.

Three shapes of the same problem

Batch serving computes predictions for a large set of instruments on a schedule — nightly, or once before market open — and stores the results in a table the trading system reads from. This is cheap, easy to test, and fine for anything with a slow-moving signal: a monthly value factor, a quarterly earnings-quality score. It is useless for anything that needs to react within the trading day, because the prediction is already hours old by the time it is used.

Online serving computes a prediction on demand, the instant a request comes in — a trading system asks "what is this model's estimate right now?" and gets an answer within milliseconds. This is what a market-making or short-horizon alpha model typically needs. The cost is architectural complexity: the model has to be loaded in memory, ready to respond, with the same feature-computation logic as training wired up to run in real time (see training-serving skew for why that wiring is the hard part).

Streaming serving sits between the two: the model is re-run continuously as new events arrive — each tick, each order book update — and it maintains and updates its own state rather than waiting for a discrete request. This suits signals that depend on a running aggregate, like an order-flow imbalance measure that needs to be recomputed on every trade print.

Worked example: choosing an architecture from a latency budget

A signal predicts 5-minute-ahead directional moves and needs to inform an order sent within 200 milliseconds of a trigger event. Suppose the full pipeline — pulling features, running the model, and returning a score — has to fit inside that 200ms window, and the trading system's own order-construction logic already consumes 120ms of it. That leaves an 80ms budget for feature computation plus inference. If feature computation from a live feed takes 50ms and a gradient-boosted tree ensemble with 500 trees takes 45ms to score, the total is 95ms — over budget by 15ms. Batch serving is not an option here (the signal is minutes old, not one 5-minute prediction is worth serving stale), so the fix has to come from inside the online path: reduce the tree count, precompute part of the feature set incrementally in a streaming layer so it's ready before the trigger fires, or move inference to a faster runtime. This is exactly the kind of arithmetic that decides architecture — not a preference for one design, but a hard deadline the model has to fit inside.

What this means in practice

Most production quant systems end up running a hybrid: slow, stable features (sector membership, market cap bucket) are computed in batch and cached, while the fast-moving inputs (current spread, recent trade imbalance) are computed online or in a streaming layer and joined with the cached values right before inference. This hybrid design is also where training-serving skew is most likely to creep in, because the batch half and the online half of the feature set are, by construction, built by different code paths on different schedules — a strong reason to test them against each other regularly rather than assume the join is correct because it compiled.

Practice

1. A model needs predictions refreshed once per trading day for a portfolio rebalance. Which architecture fits, and why would online serving be wasted engineering effort here?
2. A market-making model needs a fresh quote every time the top of book changes. Sketch, in words, why streaming rather than pure on-demand online serving is the natural fit.