Offline RL on Historical Market Data

Ordinary reinforcement learning learns by trying things: the agent takes an action, sees what happens, and adjusts. Let a trading agent do that live and "seeing what happens" means risking real capital on actions nobody has validated — an exploration phase that costs money by design. Offline RL instead trains entirely from a fixed, already-collected batch of past decisions and their outcomes — a trader's historical fills, or an existing algo's logged decisions — without the agent ever getting to try something new and see the result. It learns from a videotape, not a live rehearsal.

This solves the safety problem cleanly, but it introduces a different one: the agent can only learn about actions that show up somewhere in the data it was given. If the historical data never sized a position larger than a certain amount, the offline agent has no evidence about what would happen if it did — and it can't go find out, because there's no "going" in offline training.

Why this differs from ordinary supervised learning on historical data

Supervised learning predicts an outcome from features; if a feature combination is rare in training data, the model's prediction is just less reliable there. Offline RL is riskier in a specific way: the policy itself is being learned, and a policy that decides to take actions the training data never covers is being evaluated on data that says nothing about what those actions would have produced. The failure isn't "less reliable" — it's a genuine gap where the model has no grounding at all, and naive offline RL algorithms can produce policies that look great during training precisely because they've learned to favor actions with sparse, noisy data that happens to show good outcomes by chance.

Worked example: distributional shift

Suppose the training data comes from a historical execution algo that only ever posted limit orders at the best bid or one tick inside it — it never crossed the spread. An offline RL agent trained on this data, if it isn't constrained, can learn that crossing the spread (an action essentially absent from training) appears — by extrapolating from the few noisy examples that do exist — to produce excellent fills. Nothing in the data actually supports that; the agent is extrapolating into a region it has almost no evidence about, and confidently getting it wrong.

Conservative offline RL algorithms address this directly by penalizing the value the agent assigns to actions far from what the training data's behaviour policy actually did — explicitly discouraging the agent from confidently recommending actions it has little evidence about, rather than letting sparse noisy data masquerade as a good option.

Where it's actually useful

Offline RL is a reasonable fit when you have a large log of an existing execution algorithm's decisions and outcomes and want to learn a modest improvement on it — staying close to the logged behaviour, rather than proposing wildly novel actions, keeps the agent inside the region the data actually covers. It's a poor fit for learning a genuinely new strategy from scratch, since a genuinely new strategy by definition explores actions the historical log never took.