Inference Latency Budgets for Live Models

A model that scores brilliantly in backtests can be useless live if it's too slow to produce a prediction before the trading window it's meant to inform has closed. An inference latency budget is the maximum wall-clock time a live model is allowed to take — from receiving fresh inputs to emitting a prediction — set by the strategy's holding horizon and decision frequency, not by how fast the model happens to run today. A signal that decides whether to adjust a position every second needs its inference to complete in low milliseconds; a monthly rebalancing model can budget minutes.

The budget matters because model complexity tends to creep upward during research — an extra ensemble member here, a bigger neural network there — each improving backtest accuracy slightly, none of which shows up as a cost until the model is deployed and starts blowing through its time window. Once a model misses its budget, the choices are unattractive: skip the update and trade on stale information, or delay the decision and eat worse execution.

Concrete illustration. A market-making model has a 2ms inference budget because quotes must refresh faster than competitors can react to stale prices. During research, the team adds a gradient-boosted ensemble of 500 trees that improves out-of-sample accuracy by a small margin but takes 6ms to score on production hardware — three times the budget. The correct response is not to deploy it anyway and hope; it's to either prune the ensemble, cache intermediate features, or move the scoring onto faster hardware before the accuracy gain is allowed to matter at all.