Custom Allocators and Memory Pools

new and malloc look like a single, simple operation, but underneath they involve the general-purpose heap allocator searching for a free block of the right size, possibly asking the operating system for more memory, and updating bookkeeping structures shared across the whole program. That search takes a variable, unpredictable amount of time — usually fast, occasionally much slower, especially under memory pressure or fragmentation. For a trading system creating and destroying thousands of small objects a second (order objects, event structs, book-update messages), that variability is a direct source of latency jitter.

A memory pool sidesteps the problem by doing the expensive part once. At startup, the program allocates one large, contiguous block of memory. From then on, "allocating" an object just means handing out the next unused slot from that block — typically a pointer bump or popping an index off a free list — and "freeing" it means marking the slot available again. No system call, no searching, no fragmentation across unrelated allocation sizes.

Pool allocation versus general-purpose allocation

Worked example

An exchange simulator generates 10 million Order objects during a backtest, each 64 bytes, created and destroyed continuously as orders fill and cancel. Using new/delete for every one measured about 220 nanoseconds per allocate-deallocate pair, with a long tail — some calls take 2-3 microseconds when the allocator has to coalesce free blocks. Switching to a pool that pre-allocates 10,000 slots of 64 bytes and recycles them via a free list drops the average to about 15 nanoseconds per allocate-deallocate pair, with almost no variance — every call does the same fixed amount of work, a pointer read and a write, regardless of how many objects came before it.

The pool has a real cost too: it commits 640 KB up front (10,000 × 64 bytes) whether the simulator ever uses all 10,000 slots at once or not, and if the workload ever needs more than 10,000 live orders simultaneously, the pool must either grow (another allocation) or the design must accept the limit as a hard cap.

What this means in practice

Memory pools show up anywhere a system needs predictable, low-jitter allocation of many same-sized, short-lived objects: order objects in a matching engine, message structs in a market-data parser, node objects in a lock-free queue. They're a poor fit for objects with wildly varying sizes or long, unpredictable lifetimes, where a general-purpose allocator's flexibility matters more than a pool's speed.