Putting Transaction Costs Inside the Reward

Train an execution agent in a simulator where trades happen at the exact quoted price with no spread, no market impact, and no fees, and the agent will cheerfully learn to trade as frequently and aggressively as it likes — because in that simulator, trading is free. Deploy that same policy against a real market, where every trade crosses a spread and every marketable order moves the price a little, and the agent's learned behavior turns out to have been optimized for a world that doesn't exist. It will overtrade and bleed away exactly the costs it was never taught to avoid.

Putting transaction costs inside the reward means the simulator charges the agent, at every single training step, the same frictions it will face in production — bid-ask spread, an estimated market-impact cost proportional to trade size, and any fixed or per-share fees — so the reward the agent is trying to maximize during training is the same net quantity it will actually be judged on after deployment.

What has to be modeled

A realistic training reward typically nets out several cost components from the raw mark-to-market PnL:

• Spread cost — a marketable order pays half the spread (or more) relative to the mid, every time.
• Market impact — larger orders move the price against the trader; even a simple linear-in-size impact model (see Modelling Market Response to Agent Orders) is far better than assuming zero impact.
• Fees and rebates — exchange fees for taking liquidity, rebates for providing it, which can flip the sign of the incentive for passive versus aggressive order types.

Crucially, these costs need to scale with the agent's own actions within the simulator, not just be a flat deduction applied afterward — an agent that never sees impact cost rise as its order size grows has no way to learn that splitting a large order into smaller pieces is worthwhile.

Worked example

An agent trained with zero transaction costs in its reward learns, over 5,000 training episodes, to execute a 20,000-share order almost entirely with aggressive marketable orders in the first ten minutes, since doing so costs nothing in the frictionless simulator. Backtested against a realistic cost model after the fact, this policy incurs an average of 24 basis points per trade, mostly from front-loading into a shallow order book. Retraining the same agent with a reward that subtracts an estimated impact cost of $0.1 \times (\text{order size} / \text{average daily volume})^{0.6}$ basis points at every step produces a materially different policy: it spreads the order across the full window in smaller, mostly passive child orders. Backtested with the same cost model, its average cost drops to 9 basis points — it only learned to avoid impact because it was charged for it during training.

What this means in practice

The single most common reason a trading or execution RL agent works in simulation and fails in production is a mismatch between the reward it optimized and the true economics of trading — and transaction costs are the largest, most fixable piece of that mismatch. Before blaming the learning algorithm for poor live performance, check whether training ever actually charged the agent for the frictions it will face.