Benchmarking RL Execution Agents Against TWAP

An RL execution agent's training reward will always go up as training progresses — that's what optimization is designed to do. But "the reward went up" says nothing about whether the policy is actually good, since there's no absolute scale for a good execution cost without something to compare it to. A shortfall of 10bp sounds fine in isolation and terrible once a simple, decades-old scheduling rule achieves 6bp on the same orders with a fraction of the complexity.

TWAP (time-weighted average price) — trade equal amounts at equal time intervals across the execution window — is the standard, deliberately simple baseline every execution algorithm should be compared against, because it requires no learning, no market model, and no tuning, and yet performs reasonably well by construction, since it can't be gamed by adverse selection the way a naive aggressive strategy can.

Why the comparison has to be apples to apples

A fair benchmark requires the RL agent and the TWAP baseline to be evaluated:

• On the same set of orders — same instrument, size, and time window, ideally out-of-sample data the agent never trained on.
• Under the same cost model — if the agent's simulator has any impact or fee assumptions, TWAP must be evaluated under identical assumptions, not a more generous one.
• Across enough episodes to see the distribution, not just the average — an agent that beats TWAP on average but has a much fatter tail of bad outcomes may not actually be an improvement for a risk-averse desk, even with a better mean.

A related, slightly harder baseline is VWAP (volume-weighted average price, trading in proportion to historical volume curves rather than uniformly in time, see TWAP, VWAP & POV), which is usually a tougher benchmark to beat since it already incorporates a simple, sensible adaptation to intraday liquidity patterns.

Worked example

An RL execution agent is evaluated on 500 held-out orders against both baselines, all under an identical square-root impact cost model. TWAP achieves an average shortfall of 11 basis points (std. dev. 6bp). VWAP, benefiting from its volume-curve adaptation, achieves 8bp average (std. dev. 5bp) — a real improvement from a much simpler rule. The RL agent achieves 6.5bp average but with a std. dev. of 9bp — beating both on average, but with a noticeably wider spread, including a few orders where cost spiked above 30bp that neither baseline ever produced. That wider tail is a real finding: the desk has to decide whether a 1.5bp average improvement over VWAP is worth the extra variance, a judgment the raw training reward never surfaces on its own.

What this means in practice

No execution RL agent should go into production without a documented, apples-to-apples comparison against TWAP and VWAP on out-of-sample orders, because those simple baselines are exactly what stakeholders (and regulators, under best-execution obligations) will ask about, and because "beats a naive constant-rate strategy" is a much lower bar than "beats a sensible volume-adaptive one." A learned policy that can't clear the VWAP bar isn't obviously worth the added complexity, risk, and maintenance burden of an RL system.