Paper, Pilot and Full-Size Validation Stages

A strategy backtests beautifully and passes every validation test on historical data. Deploying it straight to full size the next day is still a bad idea, because a backtest can never fully capture what happens when a strategy meets live infrastructure, live execution, and live capital-driven market impact. Between backtest and full deployment, most serious research processes insert two intermediate gates — paper trading and a small pilot allocation — each built to catch a different class of problem that no amount of additional backtesting would reveal.

What each stage actually tests

Paper trading runs the strategy against live market data in real time, generating real signals and simulated (not real) orders, with no capital at risk. This stage tests operational correctness: does the code run reliably on the production data feed, does it handle a market holiday or a data outage or an unexpected symbol change gracefully, do the live signals match what the backtest would have produced on the same data. It cannot test anything related to actual execution quality, since no real orders are ever sent — this is its central limitation.

Pilot trading puts real capital behind the strategy, but at a small fraction of intended full size — perhaps 5-10%. This stage tests what paper trading structurally cannot: real fill quality, real slippage, real market impact at whatever size is being traded, and the psychological and operational reality of a human team managing a live position, including whether risk limits and kill switches actually fire correctly when triggered for real. A pilot that performs in line with a size-adjusted backtest expectation is the strongest evidence available, short of full deployment, that the strategy will hold up.

Full-size deployment follows only once both prior stages pass, and even then, is often scaled up gradually rather than jumped to all at once, because market impact and capacity effects that were negligible at pilot size can become material well before intended full size is reached.

Worked example: catching two different failures

A signal-generation strategy passes backtest and paper trading cleanly — the signals match exactly, no operational issues. In pilot at 8% of intended size, realized slippage runs 40% higher than the backtest's cost model assumed, because the backtest used end-of-day volume data that overstated available liquidity at the strategy's actual trade times. This is a failure paper trading structurally could not have caught (no real orders were sent) and the backtest's cost assumptions could not have caught either (they were the thing being wrong). The pilot stage exists precisely to surface this before it's discovered at full size, where the same slippage gap would be far more expensive.

What this means in practice

Skipping either stage to move faster removes exactly the test that would have caught the class of failure most likely for that particular strategy — skip paper trading and operational bugs surface with real money on the line; skip the pilot and execution-cost surprises surface at full size instead of bounded size. Define pass/fail criteria for each gate before the stage begins (e.g. "pilot slippage within 20% of backtest-modeled slippage over 100 trades"), not after seeing the results, for the same reason a lockbox test is only meaningful if the bar is set in advance.