Reinforcement Learning for Trading

Most machine learning in finance is supervised: you show the model past features and the "right answer" (up or down, buy or sell), and it learns to predict. Reinforcement learning (RL) works differently. There is no answer key. An agent takes actions in an environment, receives a reward after each one, and slowly figures out which behaviors pay off. Nobody tells it "buy here"; it discovers that buying here tended to make money, and does more of it. That framing fits trading unusually well, because a trader also gets no labels, only a running profit-and-loss statement.

The natural home for RL in markets is execution and position management, not raw price prediction. Deciding how to slice a large order over the next hour, when to cross the spread versus rest a limit order, or how to adjust inventory, these are sequences of decisions where each action changes what you see next. That feedback loop is exactly what RL is built for.

The four pieces

Every RL problem is described by a loop between an agent and its environment:

• State $s_t$ — what you observe now: your remaining inventory, time left, the order book, recent volatility.
• Action $a_t$ — what you do: buy, sell, hold, or how aggressively to quote.
• Reward $r_t$ — the immediate payoff: realized P&L minus transaction costs, or minus Market Impact.
• Policy $\pi$ — the rule mapping states to actions. Learning is the search for a good $\pi$.

The agent does not chase the biggest reward right now. It chases the biggest discounted cumulative reward over the whole episode:

$$G_t = r_t + \gamma\, r_{t+1} + \gamma^2 r_{t+2} + \cdots = \sum_{k \ge 0} \gamma^k\, r_{t+k},$$

where $G_t$ is the total future payoff from time $t$ onward, $r_{t+k}$ is the reward $k$ steps ahead, and $\gamma$ between 0 and 1 is the discount factor that says how much you care about later rewards versus sooner ones. The whole objective is to choose a policy that makes the expected $G_t$ as large as possible.

Learning the value of a move

A common approach, Q-learning, estimates $Q(s,a)$: the expected cumulative reward of taking action $a$ in state $s$ and then behaving well afterward. After each step you nudge the estimate toward what you actually saw:

$$Q(s,a) \leftarrow Q(s,a) + \alpha\Big[\, r + \gamma \max_{a'} Q(s',a') - Q(s,a)\,\Big].$$

In words: your old guess $Q(s,a)$ is corrected by a learning-rate $\alpha$ times the surprise, the reward you got plus the best value available in the next state, minus what you expected. Repeat this millions of times and the estimates converge; the policy is then "in each state, pick the action with the highest $Q$."

Worked example: one execution step

You must sell 1,000 shares over 10 minutes. State: 600 shares left, 4 minutes to go. Two actions:

• Aggressive — dump 300 shares now, crossing the spread. Immediate reward (a cost, so negative) of $−18 from impact, but it leaves you with only 300 shares and slack time, whose best continuation value $\max_{a'} Q(s',a')$ is worth $−6.
• Patient — rest limit orders, sell ~100. Immediate reward of $−4, but you still hold 500 shares into a thinning window, a continuation value of $−22.

With $\gamma = 1$, compare the totals: the aggressive move scores $-18 + (-6) = -24$, the patient move $-4 + (-22) = -26$ (all in dollars). The aggressive move looks worse on the immediate reward yet wins on the discounted total, so Q-learning steers toward it. This is the essence of what RL adds over a one-step cost model: it prices the consequences of each action on everything that follows.

Where it goes wrong

RL is seductive and brutally hard to use on markets. The usual failure is that the backtest is the environment, and the agent overfits it just as a supervised model overfits a training set, only worse, because the agent actively explores the historical data until it finds a policy that exploits its quirks.

Other traps: reward shaping that accidentally rewards the wrong behavior (maximizing raw P&L invites huge, ruinous bets, so fold risk into the reward), sample inefficiency (RL is data-hungry and financial history is short and non-repeating), and the sim-to-real gap, where a simulator that ignores your own market impact teaches a policy that stops working the moment you trade size. Used carefully, on well-posed execution and inventory problems, RL is a genuine tool; used as a black box on price prediction, it is a fast route to an overfit disaster.