Multiprocessing and Shared Memory

Python's global interpreter lock prevents two threads in the same process from executing Python bytecode at the same time, even on a machine with many CPU cores — so multithreading a CPU-heavy Python computation, like a backtest loop written in pure Python, doesn't actually speed it up. Multiprocessing sidesteps this by running separate Python processes instead of threads, each with its own interpreter and its own lock, so several processes really can run on several cores at once. This is why CPU-bound research code in Python — running many independent backtests, or many parameter combinations of the same strategy — is typically parallelized with multiprocessing rather than threading.

The catch is that separate processes don't share memory the way threads do. Threads in one process can all read and write the same variables directly; processes each get their own private copy of memory, so passing data between them means explicitly serializing it, sending it across (often through a pipe or a queue), and deserializing it on the other side. For a large dataset — say, a big price history that every worker process needs to read — copying it into every process wastes memory and time before any actual computation starts.

Shared memory is the fix for that specific problem: a block of memory that multiple processes can all access directly, without the overhead of a full copy-and-serialize step for every process. A large read-only array, like historical price data feeding twenty parallel backtest workers, can live in one shared memory segment that all twenty processes read from directly, instead of twenty separate copies each consuming memory. The trade-off is that shared memory needs careful coordination — since it bypasses each process's usual isolation, an uncoordinated write from one process can be seen inconsistently by another, so shared writable state generally needs explicit locking, the same concern that arises with threads, just avoided by default because of the extra step it takes to set shared memory up.

Worked example: parameter sweep over a large dataset

A researcher wants to backtest a strategy across 200 parameter combinations, using the same three years of price data for every run. Spawning 200 ordinary processes, each copying the full price dataset into its own memory, wastes gigabytes of redundant memory and time before a single backtest actually starts. Putting the price data into one shared memory block that all 200 worker processes read from directly avoids that duplication entirely — each process still runs its own independent parameter combination on its own core, but they all point at the same underlying data rather than 200 separate copies of it.