Parallelising a Parameter Sweep

Testing how a mean-reversion strategy behaves across ten lookback windows and ten entry thresholds means running one hundred separate backtests — a grid of every combination. Run them one after another and a two-minute backtest becomes a three-hour-and-twenty-minute wait for the whole grid, most of it spent doing nothing a researcher can use until the very last combination finishes. The hundred backtests don't depend on each other at all — combination 37 doesn't need to know the result of combination 12 — which is exactly the property that lets them run at the same time instead of in sequence.

Why independence is the whole trick

A parameter sweep parallelizes cleanly because each run is embarrassingly parallel: it reads the same input data, does its own self-contained computation, and writes its own separate result, with zero communication between runs. Contrast that with a computation where step two genuinely needs the output of step one — that has to stay sequential no matter how much hardware you throw at it. Recognizing which category a job falls into is the first move: a parameter sweep, a Monte Carlo simulation, or backtesting the same strategy across many independent stocks are all embarrassingly parallel; a single iterative optimization that updates its own state at every step generally is not.

Once you know the runs are independent, parallelizing them is mostly a matter of how many can run at once, which comes back to how many CPU cores or machines are available. Running 100 independent 2-minute jobs on 10 available cores takes roughly 20 minutes — each core works through 10 of the jobs — not 2 minutes, because the hardware caps how much genuinely happens simultaneously.

Worked example: a 100-run grid on different core counts

The 100-combination sweep above, each run taking 2 minutes: on 1 core, sequential, it's $100 \times 2 = 200$ minutes. On 10 cores, each core handles roughly $100/10=10$ runs, so the wall-clock time is close to $10 \times 2 = 20$ minutes — a 10x speedup matching the core count. On 100 cores, in principle every run happens at once and the wall-clock time approaches the 2 minutes a single run takes, though in practice some overhead (starting each job, collecting each result) keeps it a little above that floor. The speedup tracks the number of parallel workers closely as long as the individual jobs stay independent and the overhead of coordinating them stays small relative to the jobs themselves.

What this means in practice

Parallelizing a sweep changes what's practical to test at all — a grid too slow to run overnight sequentially becomes a grid you can rerun several times a day, which changes how much a researcher is willing to explore before committing to a design. The main trap is treating parallel results as if order and timing didn't matter when they secretly do: if two runs write to the same shared file, or if one run's random seed accidentally depends on when it happened to start, parallel and sequential execution can silently produce different answers. Keeping each run's inputs, outputs, and random state fully self-contained is what makes parallelizing safe rather than a source of hard-to-reproduce bugs.