Reward Design for Trading Agents

A reinforcement-learning agent does exactly what its reward function tells it to, with no sense of what you actually meant. Give a trading agent raw P&L as its reward and it will find the P&L-maximizing policy — which, with no penalty attached, is usually to take on enormous leverage and enormous risk, because the reward function never told it risk was a cost. The agent isn't malfunctioning. It's optimizing precisely what it was given, and the mismatch between "what I gave it" and "what I wanted" is the entire design problem.

Common reward mistakes, in order of how often they appear

Raw P&L with no risk term. Rewards every unit of return equally regardless of the risk taken to get it, so the agent learns to lever up. A common fix is a risk-adjusted reward — return divided by realized volatility over a recent window, closer in spirit to a Sharpe ratio than to raw dollars.

Reward only at trade close. If the agent gets zero reward every step it holds a position and only sees a number when it exits, most of its experience carries no learning signal, and training is slow and noisy. Marking P&L to market and rewarding the change each step gives a signal every period instead of only at rare exits.

No transaction cost in the reward. An agent trained without any penalty for trading learns to flip its position every step, because in a simulator with no friction, more trades never hurt and sometimes marginally help. Include realistic transaction costs and the same agent learns to trade only when the expected move clears the cost — a much more useful policy.

Reward shaping that changes the actual optimum. Adding an auxiliary reward to encourage a behaviour — a small bonus for holding through volatility, say — can, if not designed carefully, change which policy is truly optimal rather than merely making it faster to find. The theoretically safe way to add shaping without changing the optimal policy is potential-based shaping, which adds a reward equal to the change in a potential function rather than an arbitrary bonus.

Worked example: the leverage trap

An agent is trained with reward equal to daily P&L in dollars, on a simulator that allows unlimited leverage with no risk penalty. After training, it consistently takes positions many times larger than any human trader would size for the same edge, because doubling position size doubles the reward with nothing in the objective pushing back. Backtested naively — same simulator, same lack of a risk constraint — this looks like a spectacular result: enormous returns.

Change the reward to daily P&L divided by a trailing 20-day realized volatility of the portfolio, and retrain from scratch. The agent now converges to a materially smaller average position size, because oversized bets that occasionally pay off hugely also occasionally draw down hugely, and the volatility-normalized reward punishes the drawdowns as heavily as it rewards the wins. The market the agent is trading hasn't changed; only the number it's told to maximize has, and that alone moves the learned policy from something no risk desk would approve to something plausible.