Market Simulators for RL Training

A reinforcement-learning agent typically needs millions of trading decisions to converge on a good policy. No live market will let you trade millions of times to train an agent, and no historical dataset — even years of tick data — replays millions of genuinely varied scenarios; it replays one path that already happened. So RL for trading almost always trains against a simulator: a model of the market that the agent can interact with as many times as needed, cheaply and safely, before ever touching real capital.

The simulator is doing more design work than it looks like. Every regularity the agent learns is a regularity of the simulator, not necessarily of the real market — and the two are never identical.

What a simulator has to model

At minimum: how prices move in response to nothing (the baseline process), and how prices move in response to the agent's own actions (market impact). The second part is the one naive simulators skip, and it's the one that matters most for an agent learning to trade any real size — a simulator where the agent's own order never moves the price teaches it that size is free, which is false in every real market past a certain trade size.

Beyond price dynamics, a simulator needs realistic transaction costs (spread, fees, slippage that scales with size), realistic fill behaviour (a limit order doesn't always fill even when the price touches it), and ideally other market participants who react to the agent's presence rather than trading on a fixed, agent-blind script.

Two families of simulator

Historical replay feeds the agent a real historical price path and lets it trade against it. Realistic in the sense that the price dynamics genuinely happened, but the market in a replay never reacts to the agent's own orders — it plays out exactly as it did historically no matter what the agent does — so any agent that would, in reality, have moved the price with its own trading is training against a world that ignores it.

Agent-based simulation builds a market from simulated participants — market makers, noise traders, momentum followers — each with their own simple rules, and lets prices emerge from their interaction, including the trained agent's. This can react to the agent's actions, closing the gap replay leaves open, but it's only as realistic as the participant rules it's built from, and getting those rules wrong in a different way just trades one kind of unrealism for another.

Worked example: impact-blind vs impact-aware training

An agent trained on pure historical replay learns to place a large order the instant its signal fires, because in the replay, doing so never moves the price against it — the historical path is fixed regardless of the agent's order size. Deployed live, the same behaviour walks the book and the agent pays far more slippage than it saw in training; its learned policy assumed a cost that, in the real market, doesn't hold.

Retrained against a simulator with a simple square-root impact model — cost scaling with the square root of order size relative to typical volume — the agent learns to split large orders into smaller pieces over time instead of firing all at once, a behaviour the impact-blind version never had a reason to discover. The market data feeding both simulators can be identical; the only difference is whether the simulator lets the agent's own actions affect the price it trades at.