Constrained RL Under Risk Limits

A reinforcement-learning agent trained to maximize expected trading P&L has no innate concept of "never breach a $5m drawdown limit" — it only knows that breaching limits sometimes costs reward and sometimes doesn't get caught, because the training data can't cover every scenario. If you fold the risk limit in as just another term subtracted from the reward, the policy learns to trade off limit breaches against profit exactly like any other cost, meaning a big enough expected gain can still make it worth risking a breach. That's unacceptable for a hard risk limit, which by definition is not supposed to be negotiable. Constrained RL is the framework for training a policy that maximizes reward subject to limits it is never allowed to cross, rather than merely discouraged from crossing.

The analogy: a speed limit versus a toll

A toll road charges you money for using it — more tolls, less profit, but you can always choose to pay if the trip is worth it. A speed limit is different: it's not a price you can pay to go faster, it's a line you are not supposed to cross regardless of how much time you'd save. Standard reward-shaping treats a risk limit like a toll — a penalty you can outweigh with enough reward. Constrained RL treats it like a speed limit — a separate constraint the policy must satisfy, evaluated on its own, independent of how attractive breaching it might look on the reward side.

Formally, instead of training the policy to maximize expected reward alone, constrained RL maximizes expected reward subject to keeping a cost signal — say, expected drawdown or portfolio VaR — below a fixed threshold. During training this is usually enforced via a Lagrangian penalty whose weight is automatically increased whenever the policy drifts toward violating the constraint and relaxed once it's safely inside the limit, so the policy learns to satisfy the constraint reliably rather than merely on average.

Worked example: a position-cap trader

Suppose an agent trades a single futures contract and must never hold more than 50 contracts of net exposure, with a reward of realized P&L each step. An unconstrained agent, if it discovered that briefly holding 80 contracts around a strong signal boosted average reward, would learn to do so — the position cap only exists in the training data as "sometimes profitable, sometimes not." A constrained version tracks a cost signal equal to how far current position exceeds the 50-contract cap, and trains the policy to keep expected constraint violation near zero. In practice this shows up as the policy learning to scale down its trade size as it approaches 50 contracts, effectively self-imposing a soft buffer well before the hard cap, because breaching it — not just profit lost from breaching it — is penalized as a constraint failure the optimizer treats as categorically different from ordinary reward.

What this means in practice

Any RL trading system that has to sit inside a firm's actual risk framework needs this distinction, because real risk limits are not negotiable trade-offs — a drawdown limit exists precisely so no amount of expected future profit is allowed to justify breaching it in the moment. Constrained RL is also what makes an RL policy auditable by a risk desk: instead of hoping the reward function was shaped just right, a risk manager can point to an explicit constraint and a measured violation rate and ask for it to be tightened.