On-Policy vs Off-Policy Learning
Whether a reinforcement-learning algorithm must learn from the actions its own current policy actually took, or can learn from any past data regardless of which policy generated it.
Prerequisites: Q-Learning, The Policy Gradient Theorem
Every reinforcement-learning algorithm has to answer a basic question: whose experience is it allowed to learn from? On-policy algorithms can only learn from actions generated by the exact policy currently being trained — if the policy changes even slightly, the old data is considered stale and effectively has to be collected again. Off-policy algorithms can learn from actions generated by a different policy entirely — an older version of themselves, a hand-built rule, even pure exploration — and still correctly figure out how the policy being trained would have performed.
Q-learning is the classic off-policy example: it updates its estimate of "the best possible value of taking this action" using the best next action available, regardless of what action the data-collecting policy actually took next. That means a Q-learning agent can train on a giant archive of historical trades logged under an old strategy, or even random exploratory trades, and still learn a good policy. SARSA is the on-policy counterpart: it updates using the value of the action the current policy actually would take next, which means the value estimates are only accurate for the policy that generated the data, and go stale the moment the policy changes.
The trade-off is real, not just technical. On-policy methods tend to be more stable because they're always evaluating the policy they're actually improving, with no mismatch to correct for. Off-policy methods are far more data-efficient because they can reuse old data (replay buffers), but they need to correct for the mismatch between the policy that generated the data and the policy being evaluated, which gets harder and noisier the more the two policies differ.
What this means in practice
For trading applications this distinction decides whether you can reuse historical execution logs to train a new policy or whether you're stuck collecting fresh data every time the policy changes even slightly — a difference of months versus days in how fast a strategy can iterate. It's also exactly the question at the heart of off-policy evaluation: estimating how a brand-new trading policy would have performed using only logs from an old one, without ever running the new policy live.
On-policy learning only trusts data from its own current policy and goes stale whenever the policy changes; off-policy learning can reuse data from any policy, at the cost of needing to correct for the mismatch between the data-generating policy and the one being trained.
Related concepts
Practice in interviews
Further reading
- Sutton & Barto, Reinforcement Learning: An Introduction, ch. 5-6