Known-Answer Tests for Simulators

A backtesting simulator is itself a piece of software, and like any software it can have bugs — a mis-handled corporate action, an off-by-one error in matching fills to timestamps, a fee calculated on the wrong side of a trade. These bugs are dangerous precisely because they don't crash the program; they just quietly produce a plausible-looking but wrong number, and a wrong Sharpe ratio looks exactly like a right one until you dig in.

A known-answer test sidesteps this by constructing a scenario simple enough that you can compute the correct output by hand (or with a trivial spreadsheet) before ever running it through the simulator. If the simulator's output matches your hand calculation, that piece of its logic is validated; if it doesn't, you've caught a bug before it contaminated results on a real strategy where the correct answer isn't known in advance.

Worked example. To test a simulator's handling of trading costs, construct a toy scenario: buy 100 shares at $10.00 with a flat $1 commission and a 5 basis point slippage assumption, then sell them a day later at $10.50 with the same costs. By hand, the trade should show a gross gain of $50, minus $2 in commissions and roughly $1.03 in slippage (5bps of each $10.25 average notional), netting to about $46.97. If the simulator reports anything else, the bug is in the cost model, not in the strategy being tested.