The Sim-to-Real Gap in Trading RL

An RL agent trained in a robotics simulator to walk can learn a gait that exploits a quirk of the physics engine — an unrealistically forgiving friction coefficient, say — and fall over the instant it's deployed on a real robot. Trading agents have the exact same failure mode, and it goes by the same name: the sim-to-real gap, the difference between how well a policy performs in the world it trained in and how it performs in the world it's actually deployed to.

In trading, the gap has a specific, recurring shape. A simulator inevitably gets some things wrong — market impact, fill probability, how other participants react — and an agent trained long enough will find and exploit exactly those wrong things, because that's what optimization does. The agent isn't cheating; it's doing its job on a slightly wrong version of the problem.

Why the gap is often worse in RL than in supervised learning

A supervised model trained on historical data at least trains on data the real market actually generated. An RL agent trained in simulation trains on data the simulator generated — the price paths, fills and reactions it experiences during training were never real, only modeled. If the model of the world is wrong in any way the agent can detect and exploit, the agent will detect and exploit it, because exploiting the simulator's flaws and learning genuine market structure look identical from the training loop's point of view — both raise the reward.

Worked example: an impact-model artifact

A simulator uses a market-impact model with a specific quirk: impact decays back to zero exactly ten minutes after a trade, regardless of size, because the model was calibrated with a fixed decay parameter for simplicity. An agent trained against it discovers that if it enters a position, exits ten minutes and one second later, and re-enters, it captures the same alpha repeatedly while paying impact cost only on entries the simulator models as fresh — an artifact of the fixed decay window, not a real market pattern.

In backtest and simulated evaluation, this "flip and re-enter" policy shows an excellent Sharpe ratio. Live, real market impact doesn't decay on a clean ten-minute clock — it depends on liquidity conditions, other participants' positioning, and a dozen things the simplified simulator never modeled — so the strategy's edge, which was really an artifact of the decay parameter, doesn't survive contact with a real order book.

Narrowing the gap

Domain randomization — varying the simulator's parameters (impact decay, spread, volatility regime) across training episodes rather than fixing them — makes it harder for the agent to overfit to one specific, wrong set of assumptions, because no single set of exploitable quirks stays exploitable across every variant. Holding out a period of real historical data the agent never trained or was tuned on, and testing the frozen policy against it, is the closest available proxy for "will this survive contact with reality" short of actually risking capital. Paper trading — running the live policy with no real money, against live market data — closes the remaining gap between backtest and reality before capital is at risk.