Offline Reinforcement Learning

Every reinforcement-learning method covered so far — Q-learning, REINFORCE, actor-critic — assumes the agent gets to keep acting in the real environment, trying new things and observing what happens next. That assumption is often false in finance: you typically have a large, fixed archive of past trading data — historical prices, historical fills, historical order-book snapshots — generated by whatever strategy or strategies were actually running at the time, and no ability to go back and try a different action on a historical Tuesday to see what would have happened instead. Offline reinforcement learning (also called batch RL) is the branch of the field built specifically for this situation: learn the best possible policy from a fixed, already-collected dataset, with zero further interaction with the real environment allowed during training.

Why this is harder than it sounds

The core difficulty is that a fixed dataset only ever shows you the consequences of the actions that were actually taken when it was collected — never the consequences of any alternative action the historical policy did not happen to try. An offline RL algorithm trying to evaluate "what if I had done something different in this state" has to extrapolate, using a value function or model, into action choices the data never actually demonstrates the outcome of. This is fundamentally different from online RL, where the agent can simply go try the alternative action itself and see. The risk is a kind of overconfident extrapolation: a naive method can end up assigning a high value to actions the historical data almost never took, purely because nothing in the dataset contradicts an overly optimistic guess about them, and then confidently recommend a policy built substantially on that guess.

Worked sketch: overestimating an untried action

Suppose a historical dataset was generated by a policy that, in a particular state (high realized volatility, holding a long position), almost always chose "hold" and occasionally "reduce," but essentially never "add to position." A naive value-based method trained on this data still has to estimate a value for "add to position," since a complete value function needs an estimate for every action — and with almost no data showing what happens after "add," that estimate is built mostly on extrapolation. If the function happens to extrapolate optimistically, the resulting policy can end up recommending "add to position" heavily, specifically because it has so little real evidence to be corrected by. Deployed live, that recommendation meets a real market for the first time, and the never-actually-observed consequences can turn out badly. Modern offline RL methods address this by explicitly penalizing value estimates for actions far outside what the data supports, staying conservative exactly where the data offers no evidence.

The overshoot behavior a naive offline method exhibits is visually similar to the overshooting trajectory in a picture like the one above: confidently moving in a direction that looks good locally (per the fitted value function) but that has never actually been tested, with nothing available to correct course before real consequences arrive.

What this means in practice

Offline RL is the honest framing for most "backtest a policy on historical data" trading work, since a genuine live-interaction reinforcement-learning loop against real markets is usually too slow, too risky, or simply unavailable during development. The field's main practical lesson for a quant is to be deliberately conservative about any policy recommendation that deviates sharply from what the historical data actually did — the further a proposed action sits from the historical policy's typical behavior, the less the training data can actually vouch for it, regardless of how confident the fitted value function looks.