Inference Latency Budgets

In a general web service, a latency budget is measured in milliseconds and a slow request is an annoyance. In a trading system reacting to a tick of market data, the same idea applies to a budget measured in microseconds, because the whole point of the model is to act before the price the model was trained to predict has already moved. Inference latency budgeting in a quant-dev context is dividing that microsecond-scale window — from market event to order sent — among feature computation, model evaluation, and risk checks, and treating any step that blows its share as a correctness bug, not just a performance nuisance.

The analogy: a pit stop measured in tenths of a second

A Formula 1 pit crew doesn't have a vague goal of "change the tires quickly" — every task (jack up the car, remove old tire, fit new tire, lower the car) has a rehearsed time slice within a roughly 2-second total, and a single team member running half a second over their slice costs the whole race position. A trading system's tick-to-trade path is the same discipline applied to computation: feature extraction, model scoring, and risk checks each get a slice of a budget measured in single-digit microseconds, and a stray heap allocation blowing past its slice is exactly like a pit-crew member fumbling a bolt.

Worked example 1: the tick-to-trade budget

A market-making system has a total tick-to-trade budget of 10 microseconds (μs) from receiving a book update to sending a quote update. A candidate breakdown:

Step	Budget
Parse the market data message	1.5μs
Compute features (spread, imbalance, short-term momentum)	3.0μs
Evaluate the model (a small gradient-boosted tree ensemble)	3.5μs
Risk checks (position limits, fat-finger checks)	1.0μs
Serialize and send the order	1.0μs

These sum to exactly 10μs. If profiling shows model evaluation actually takes 5μs — 1.5μs over budget — the fix is architectural, not "try harder": reduce the model (fewer trees, shallower depth), or move the risk checks to run in parallel with the last stage of feature computation instead of strictly after it.

Worked example 2: why a garbage collector is a budget-breaker, not a rounding error

Suppose a system written in a garbage-collected language usually evaluates its model in 2μs, comfortably inside the 3.5μs slice above. But once every few thousand ticks, a garbage collection pause adds 200μs — twenty times the entire tick-to-trade budget, all at once. Across 100,000 ticks a day, even a 1-in-5,000 pause rate means roughly 20 ticks a day where the system is effectively 20x too slow to react before the market moves. That single, rare stall is why performance-critical inference paths are commonly written in C++ with careful, preallocated memory rather than a garbage-collected language — the budget has to hold on essentially every tick, not just on average.

Treat the curve above as a stand-in for tail latency versus load: near the flat part, adding a little more work barely changes response time, but past a threshold — a queue starting to build, a cache starting to miss — the same small increase in load produces a runaway increase in latency, which is exactly the shape a tick-to-trade budget is trying to stay well clear of.

What this means in practice

Quant developers profile every stage of the tick-to-trade path individually, in the worst case as well as the typical case, and choose languages, memory strategies, and even model architecture with that budget as a hard constraint rather than an afterthought. A model that would be a fine choice in an ordinary ML latency budget — a modestly large neural network, say — can simply be disqualified here purely on evaluation-time grounds, regardless of its accuracy.