Framing a Trading Problem as an MDP

Before you can apply any reinforcement-learning algorithm to a trading decision, you have to answer a question that has nothing to do with the algorithm: what, exactly, is the state the agent sees, what actions can it take, and what reward does it get? Get this framing wrong and the fanciest algorithm in the world optimizes the wrong problem perfectly.

A Markov decision process, or MDP, is the formal container for "state, action, reward, repeat." It assumes the current state contains everything relevant to deciding the next action — the Markov property — so the agent doesn't need to remember the whole history, just where things stand right now. See Markov Decision Processes for the general machinery; this page is about the harder, less mechanical question of fitting a trading problem into that container in the first place.

The three choices

State: what does the agent get to see before deciding? Current position and cash are obvious inclusions. Less obvious: how much recent price history, whether order-book depth is visible, whether the agent knows its own past actions' market impact. Include too little and the agent can't distinguish situations that genuinely call for different actions — it violates the Markov assumption in practice even if the label is on it in theory. Include too much and you've handed the agent a training problem that needs far more data than you have to learn from.

Action space: a simple framing is three actions — buy, hold, sell, in fixed size. This is easy to learn but throws away information; real execution decisions are about size and timing, not just direction. A richer framing — choose a position size in some range, or place a limit order at some price offset — is more realistic and much harder to train, because the agent now has to learn a continuous decision instead of pick from three buttons.

Reward: what number does the agent actually get after each action, and does maximizing it correspond to what you actually want? This turns out to be the single easiest place to get a trading MDP wrong, and it gets its own page — see Reward Design for Trading Agents.

Worked example: two framings of the same problem

Consider an agent deciding whether to hold or exit a mean-reverting spread position. Framing A: state is just the current spread z-score; actions are {enter, hold, exit}; reward is realized P&L on exit, zero otherwise. This is Markov-clean and trains fast, but the agent never sees how long it has been in the position, so it can't learn that a spread stuck away from zero for thirty days behaves differently than one on day two — a real risk, since stuck spreads often signal a broken relationship rather than a slow reversion.

Framing B: state adds days-in-position and a rolling measure of spread volatility; reward is marked-to-market P&L each period rather than only at exit, so the agent gets a learning signal every step instead of only at the rare moments it closes a trade. Framing B trains slower — the state space is bigger — but produces an agent capable of learning "cut this if it's been stuck for two weeks," a rule Framing A cannot represent no matter how long you train it.

Where this goes wrong in practice

The most common mistake is smuggling information into the state that wouldn't actually be available at decision time — using a feature computed with data that arrives after the trade would have to be placed. This is the same lookahead problem that plagues supervised backtests, and an MDP framing hides it just as easily since the state is assembled by the same pipeline code.