Inference Latency Budgeting
Serving a model in production means dividing one fixed time budget among every step a prediction must pass through, and the step that matters is the slowest one you'll ever hit, not the average.
Prerequisites: Model Monitoring in Production
A model that scores in 2 milliseconds in a notebook can still miss its production deadline. The reason is that a real prediction is never just "run the model" — it is fetch features, run the model, post-process the output, and ship a response back over the network, and every one of those steps eats into a fixed time budget the business has already promised somewhere downstream: a web page that must render in 200ms, a trading system that must quote before the market moves. Inference latency budgeting is the discipline of splitting that one fixed total among the steps, and deciding in advance which step gets cut when something goes over.
The analogy: a relay race with a fixed finish time
Imagine a relay team told the whole race must finish in 40 seconds, split across four runners. You would not let each runner train independently and hope the sum works out — you'd assign each leg a target (say 8, 12, 10, 10 seconds) that sums to the deadline, and if one runner is consistently the slowest, either that leg gets a stronger runner or the other legs give up time to cover the gap. A latency budget is that same allocation, applied to feature retrieval, model inference, and network transit instead of runners.
Worked example 1: building the budget
A recommendation service has an SLA of 100ms end-to-end. Break it down:
| Step | Budget |
|---|---|
| Feature fetch (database/cache) | 40ms |
| Model inference | 30ms |
| Post-processing (ranking, filtering) | 15ms |
| Network round-trip | 15ms |
These sum to exactly 100ms, with no slack. If feature fetch measures 55ms in production — 15ms over its allotment — the team has two choices: speed up the fetch (cache more aggressively) or take 15ms from another line item, most easily inference, by switching to a smaller or quantized model. The budget makes the trade-off visible instead of it being an unowned surprise at 3am.
Worked example 2: the average lies, the tail decides
Suppose the inference step measures a mean of 8ms across a day of traffic. That sounds comfortably inside its 30ms allotment. But look at the full distribution: 99% of requests take 6–10ms, while 1% take 90ms because of a garbage-collection pause or a cold cache. If the service handles 10,000 requests a minute, that "rare" 1% is 100 requests a minute blowing the entire 100ms SLA — every minute, reliably. Budgets are set and monitored against p99 (the 99th-percentile latency), not the mean, precisely because the mean hides exactly the events that break the deadline.
The gap between mean and tail above is worth seeing on a real distribution, not just a bar chart. Drag the shape below and watch how a long right tail pulls the 99th percentile far past where the bulk of the mass — and the mean — actually sits.
A latency budget allocates a fixed total across pipeline steps so trade-offs are decided on paper, before an outage — and it must be set against the tail (p99, not mean latency), because tail events are what breach an SLA in practice.
What this means in practice
Teams instrument every step with percentile latency (p50, p95, p99) and alert on the tail, not the average. When a budget is broken, the fix is rarely "make the model faster" in the abstract — it is a specific trade against a specific line item: quantize or distill the model to shrink inference time, batch requests to amortize overhead, or cache features to avoid a slow database call. The budget is what turns "the system feels slow sometimes" into an assignable, fixable number.
The classic mistake is budgeting and monitoring the mean latency instead of the tail. A system can have an excellent average latency and still violate its SLA for a meaningful fraction of users every single day, because rare slow requests (cache misses, GC pauses, cold model instances) are exactly the ones a mean silently averages away. Always budget and alert against p95/p99, and treat "average latency looks fine" as no evidence at all about tail behavior.
Related concepts
Practice in interviews
Further reading
- Crankshaw et al., Clipper: A Low-Latency Online Prediction Serving System (2017)