Latency Percentiles and Tail Latency

An engineer reports "our order gateway's average latency is 50 microseconds," and a trader shrugs and asks the wrong follow-up question, because average latency almost never determines whether a strategy makes or loses money. What matters is how often the system is slow, because in trading, being slow at the wrong moment, the exact instant a price is about to move, is where the money is lost. A gateway that's fast 999,999 times out of a million and catastrophically slow the one time that matters can lose more than one that's mediocre but consistent every single time.

The fix is to stop summarizing latency with a single number and instead report percentiles: sort every measured latency from fastest to slowest, and the $p$-th percentile is the value below which $p\%$ of measurements fall. The median ($p50$) is the typical case. The interesting numbers live further out: $p99$ (the slowest 1% starts here), $p99.9$, and $p99.99$, collectively the tail. A system can have a beautiful median and a nightmarish tail, and the two numbers are close to independent: nothing about a fast median guarantees anything about the tail, because they're typically driven by entirely different causes, a fast median reflects the steady-state hot path, while a bad tail reflects rare events like a garbage collection pause, a cache eviction, an OS scheduling hiccup, or lock contention with another thread.

Worked example: mean hides the tail

An order gateway logs 100,000 round-trip times. 99,000 of them take 10 microseconds; 900 take 100 microseconds (occasional cache misses); 100 take 5 milliseconds (rare garbage-collection or OS scheduling pauses). The mean is:

That 15.8 μs mean sounds excellent, and it's barely above the 10 μs typical case. But $p99$ sits right at the boundary between the 100 μs and 5 ms groups, roughly 100 μs, and $p99.9$ lands inside the 5 ms group entirely. So while the average order takes 15.8 μs, the slowest 1-in-1,000 orders take over 300x longer than typical, a fact the mean completely conceals. If a strategy's edge depends on being fast in exactly the moments the market is moving fastest, precisely when contention and pauses are most likely, the p99.9 number is what determines whether that edge survives, not the mean.

What this means in practice

Latency-sensitive systems are designed and monitored against tail percentiles, not averages: SLAs quote "p99.9 under 200 μs," not "average under 50 μs." Improving the tail usually requires different fixes than improving the median, removing garbage collection, pinning threads to isolated cores, avoiding lock contention, because the tail is caused by rare events the median never touches. Reducing the median while ignoring the tail can make a system look better on a dashboard while actually making its worst-case behavior, the behavior a strategy's risk depends on, no better at all.