Latency Percentiles and Tail Latency

An engineer reports "our order gateway's average latency is 50 microseconds," and a trader shrugs and asks the wrong follow-up question — because average latency almost never determines whether a strategy makes or loses money. What matters is how often the system is slow, because in trading, being slow at the wrong moment — the exact instant a price is about to move — is where the money is lost. A gateway that's fast 999,999 times out of a million and catastrophically slow the one time that matters can lose more than one that's mediocre but consistent every single time.

The fix is to stop summarizing latency with a single number and instead report percentiles: sort every measured latency from fastest to slowest, and the $p$-th percentile is the value below which $p\%$ of measurements fall. The median ($p50$) is the typical case. The interesting numbers live further out: $p99$ (the slowest 1% starts here), $p99.9$, and $p99.99$ — collectively the tail. A system can have a beautiful median and a nightmarish tail, and the two numbers are close to independent: nothing about a fast median guarantees anything about the tail, because they're typically driven by entirely different causes — a fast median reflects the steady-state hot path, while a bad tail reflects rare events like a garbage collection pause, a cache eviction, an OS scheduling hiccup, or lock contention with another thread.

Worked example: mean hides the tail

An order gateway logs 100,000 round-trip times. 99,000 of them take 10 microseconds; 900 take 100 microseconds (occasional cache misses); 100 take 5 milliseconds (rare garbage-collection or OS scheduling pauses). The mean is:

$$\bar{x} = \frac{99{,}000 \times 10 + 900 \times 100 + 100 \times 5000}{100{,}000} = \frac{990{,}000 + 90{,}000 + 500{,}000}{100{,}000} = 15.8 \; \mu s.$$

That 15.8 μs mean sounds excellent, and it's barely above the 10 μs typical case. But $p99$ sits right at the boundary between the 100 μs and 5 ms groups — roughly 100 μs — and $p99.9$ lands inside the 5 ms group entirely. So while the average order takes 15.8 μs, the slowest 1-in-1,000 orders take over 300x longer than typical, a fact the mean completely conceals. If a strategy's edge depends on being fast in exactly the moments the market is moving fastest — precisely when contention and pauses are most likely — the p99.9 number is what determines whether that edge survives, not the mean.

What this means in practice

Latency-sensitive systems are designed and monitored against tail percentiles, not averages: SLAs quote "p99.9 under 200 μs," not "average under 50 μs." Improving the tail usually requires different fixes than improving the median — removing garbage collection, pinning threads to isolated cores, avoiding lock contention — because the tail is caused by rare events the median never touches. Reducing the median while ignoring the tail can make a system look better on a dashboard while actually making its worst-case behavior — the behavior a strategy's risk depends on — no better at all.