Jitter and Determinism

Two market-data handlers both average 10 microseconds of processing time per tick. One always takes between 9 and 11 microseconds. The other usually takes 5 microseconds but occasionally spikes to 200 microseconds. Same average, wildly different systems — the second one is unusable for anything timing-sensitive, because you can never trust its next reading. That spread, run to run, is jitter: the variability in how long an operation takes, independent of how long it takes on average. A system with low jitter is deterministic — not in the sense of always doing the exact same thing, but in the sense that its timing is predictable, tightly clustered around a known value rather than scattered.

Jitter comes from sources that have nothing to do with the algorithm itself: the operating system scheduler pausing a thread to run something else, a cache line getting evicted by a neighboring process, a garbage collector pausing execution to reclaim memory, dynamic CPU frequency scaling changing clock speed mid-task, or a memory allocator occasionally needing to request more memory from the OS. None of these show up in a clean, single-threaded benchmark — they show up under real, contended, multi-process production load, which is exactly why jitter is so easy to miss in testing and so damaging in production.

Worked example: why jitter beats raw speed for market making

A market maker's quote-update logic must react to a price change and requote before a faster competitor does. System A processes updates in a consistent 8–12 microseconds. System B is on average faster — 6 microseconds — but has a long tail: 1 in 200 updates takes over 150 microseconds because of an occasional garbage-collection pause. Over a trading day with roughly 2 million price updates, System B stalls badly on about 10,000 of them. Each stall is a window where a competitor with a consistent 8–12 microsecond system can pick off a stale quote before System B updates it — meaning System B's better average case actively loses money on its worse tail case, at a rate proportional to how often the stall coincides with a real price move. System A's higher but tightly-bounded latency is worth more in practice than System B's lower but jittery one, because a market maker can size its quotes around a known worst case, but can't safely size around an unpredictable one.

What this means in practice

Engineering for determinism means actively removing sources of variability, not just shrinking the average: pinning threads to isolated CPU cores so the OS scheduler can't interrupt them, avoiding memory allocation (and hence garbage collection) on the hot path, disabling CPU frequency scaling on latency-critical cores, and busy-polling instead of relying on interrupt-driven wakeups whose timing the OS controls. Each of these can make the typical case look slightly worse — busy-polling burns a full core doing nothing most of the time — in exchange for a much tighter, more trustworthy worst case, which is usually the better trade for latency-sensitive trading systems.