Market Data Compression

A year of full-depth order book data for a few hundred liquid instruments can run into terabytes uncompressed. Storing that raw is expensive and, worse, slow to read back — a backtest that scans a year of history spends most of its time waiting on disk I/O, not computing. Compression addresses both problems at once: smaller files cost less to store and, because reading fewer bytes is often faster than reading more even after paying the cost to decompress them, well-compressed data can make a backtest run faster, not slower.

Two properties of market data make it unusually compressible compared to arbitrary bytes. First, timestamps in a sorted tick stream are almost always very close to the previous timestamp — the difference (the "delta") between consecutive ticks is a small number even when the timestamps themselves are large, so storing deltas instead of absolute values shrinks the data dramatically before any general-purpose compression is even applied. Second, prices move in small increments most of the time; a price series is far more compressible than an equivalent stream of independent random numbers, because a compressor can exploit the fact that most values are close to their neighbors.

Compression sits on a real trade-off between size and speed, and the right choice depends on how the data will be used. A very aggressive compression algorithm shrinks files the most but is slow to decompress, which is fine for cold archival data rarely touched again but painful for data read constantly during backtest iteration. A lighter, faster algorithm decompresses quickly but leaves more bytes on disk. Systems commonly use different settings for different tiers: light, fast compression on hot recent data queried often, heavy compression on cold historical archives queried rarely.

Worked example: delta encoding a tick stream

Raw timestamps for five consecutive ticks (in milliseconds since a session start): 100000, 100003, 100003, 100011, 100014. Stored as-is, each needs enough bits to represent a number up to 100014. Delta-encoded — storing the first value, then the difference from the previous one — the sequence becomes: 100000, 3, 0, 8, 3. After the first value, every number fits in a handful of bits instead of needing enough range for six-digit numbers, because deltas between real-world tick arrivals are almost always small. This single transformation, applied before any general compression algorithm runs, is often responsible for most of the size reduction in a tick data store.

What this means in practice

A backtest or research pipeline that reads a lot of historical tick data benefits directly from a well-compressed store — read time often dominates iteration speed more than compute time does. Live systems are more cautious about heavy compression on the hot path, since decompression adds latency exactly where latency matters most; live feed handlers typically keep only the lightest compression, or none, on data that needs to be processed within microseconds.