PnL vs Risk-Adjusted Reward Functions

Tell a reinforcement-learning agent that its only goal is to maximize total profit, and it will find the mathematically optimal way to do exactly that — which is rarely the way a risk-conscious trading desk would want. An agent rewarded purely on raw PnL has no built-in reason to avoid a strategy that makes steady small gains for months and then occasionally blows through a catastrophic loss, as long as the average profit over training episodes comes out ahead. The agent is optimizing exactly what it was told to optimize; the problem is that raw PnL was the wrong thing to tell it.

Risk-adjusted reward functions replace or augment raw PnL with a measure that explicitly penalizes variance, drawdown, or tail risk, so the agent's incentive structure actually matches what a real trading operation cares about: consistent, controllable performance, not just the highest expected payoff regardless of how it's achieved.

Common risk-adjusted reward formulations

• Differential Sharpe ratio — an incremental, per-step approximation of the Sharpe ratio that can be computed online rather than only at the end of an episode, giving the agent immediate feedback that rewards consistency, not just cumulative return.
• Drawdown-penalized reward — raw PnL minus a penalty term proportional to the current drawdown from the equity curve's running peak, directly discouraging the agent from strategies that produce deep troughs even if they recover.
• Downside-deviation-based reward (Sortino-style) — penalizing only the variance of negative returns, leaving upside volatility unpunished, since an agent shouldn't be discouraged from occasional large gains, only large losses.

Each of these changes what the agent is being told to optimize, not just how it's trained — the same learning algorithm applied to raw PnL versus a drawdown-penalized reward can converge to qualitatively different policies.

Worked example

Two identical agents train on the same simulated market with the same algorithm, differing only in reward. Agent A is rewarded purely on daily PnL; Agent B on daily PnL minus $0.5 \times$ current drawdown from the equity peak. Agent A converges to a policy earning a higher average annualized return of 22%, but with a maximum drawdown of 38% and a Sharpe of 0.9 — it discovered that large concentrated positions raise average profit despite occasional steep losses. Agent B converges to a lower 15% return, but with only a 14% maximum drawdown and a Sharpe of 1.6 — the penalty pushed it toward smaller, diversified sizing. Which agent is "better" depends entirely on the desk's risk tolerance, which is exactly the judgment the reward function is meant to encode before training begins.

What this means in practice

The reward function is a specification of the desk's actual objective, and skipping straight to raw PnL is usually a mistake born of simplicity, not correctness — no real trading operation actually wants "maximum average profit with no regard to variance." Because these choices interact with position sizing and transaction costs, risk-adjusted reward design is usually paired with explicit cost modeling in the reward itself (see Putting Transaction Costs Inside the Reward) rather than tackled in isolation.