Measuring Latency: Percentiles And Tails

Suppose a trading system reports "average round-trip latency: 80 microseconds." That sounds fast. But if one message in a thousand takes 50,000 microseconds because of a garbage collection pause or a network retransmit, the average barely moves — a single huge number spread across a thousand samples adds only 50 microseconds to the mean — while that one message arrives so late it trades on a stale price and loses money on every occurrence. The average is the wrong question. The right question is: how bad is the worst case, and how often does it happen?

Think of it like commute times. If your commute averages 25 minutes but has a 1-in-100 chance of taking two hours because of an accident, "average 25 minutes" is a useless number for deciding when to leave for a flight. You care about the tail: the 99th percentile commute time, because that's the one you have to plan around.

A percentile answers "what value is X% of observations at or below." The median is the 50th percentile. In latency work the numbers that matter are far out: p99 (the value 99% of messages beat), p99.9, and often p99.99 — because in a system processing millions of messages a day, even a "rare" 1-in-10,000 event happens hundreds of times.

Worked example. Take 20 round-trip latency samples, in microseconds, sorted: 40, 42, 43, 45, 45, 46, 48, 49, 50, 51, 52, 53, 55, 58, 60, 65, 72, 95, 210, 1800

The mean is $\frac{40+42+\dots+1800}{20} = \frac{3009}{20} = 150.5\,\mu s$ — dragged up almost entirely by the single 1800 outlier. The median (average of the 10th and 11th sorted values, 51 and 52) is $51.5\,\mu s$ — a much more honest picture of the "typical" trade. To find the p95, take rank $\lceil 0.95 \times 20 \rceil = 19$, the 19th sorted value: $210\,\mu s$. The p99 needs rank $\lceil 0.99 \times 20 \rceil = 20$, the very last value: $1800\,\mu s$. With only 20 samples the p99 is just the max, which is exactly the problem with small samples — you need thousands of observations before a p99 or p99.9 estimate is stable.

Second example, at scale. A venue gateway processes 1,000,000 messages in a day. Its measured latencies: median 60 μs, p99 = 400 μs, p99.9 = 3,000 μs, p99.99 = 45,000 μs (45 ms). How many messages fall in that worst 0.01% bucket? $1{,}000{,}000 \times 0.0001 = 100$ messages — a hundred trades a day arriving 45 milliseconds late, which in a fast-moving market is enough time for the price to have moved several ticks. If each of those 100 late fills costs an average of $8 in adverse slippage versus a fast fill, that tail alone costs $800/day, or roughly $200,000/year trading days — money that never shows up if you only ever look at the mean.

What this means in practice

Latency SLAs (service-level agreements) on trading infrastructure are written in percentiles, never means, for exactly this reason: "p99.9 under 500 microseconds" tells an engineer precisely which tail to hunt down, while "average under 100 microseconds" can be satisfied by a system that is fast 99% of the time and catastrophically slow the rest. Tail latency usually has an identifiable cause — a garbage collection pause, a lock contention spike, a network switch buffer filling up — and fixing that one cause moves the p99.9 far more than any amount of average-case tuning.

The classic confusion

Latency percentiles are the natural companion to Latency vs Throughput — a system can trade off the two, and the percentile view is what tells you whether a throughput gain came at the cost of a fatter tail. See Modelling Latency In Backtests for how these distributions get simulated rather than measured live.