Trust Region Policy Optimization

Ordinary policy-gradient methods update a trading or control policy by taking a step in the direction that seems to improve expected reward, using a chosen step size. The problem is that a step which looks small in parameter space can correspond to a huge, destabilizing change in actual behavior — a tiny nudge to a neural network's weights can flip a policy from "hold" to "sell everything," and once the policy has moved that far, all the past data used to estimate the gradient is stale, and performance can collapse and never recover.

Trust Region Policy Optimization (TRPO) fixes this by explicitly constraining how much the new policy is allowed to differ from the old one at each update, measured not in raw parameter distance but in KL divergence — a measure of how different two probability distributions over actions are. The update maximizes expected improvement subject to that KL divergence staying inside a small "trust region":

$$\max_\theta \; \mathbb{E}\big[\text{advantage of new policy over old}\big] \quad \text{subject to} \quad \mathrm{KL}(\pi_{\text{old}} \,\|\, \pi_\theta) \le \delta$$

In words: pick the update that improves expected performance the most, but never allow the new policy's action probabilities to drift further from the old policy's than a small allowed budget $\delta$. This guarantees each step is a genuine, provable improvement rather than a gamble.

The cost is complexity: solving that constrained optimization exactly requires a second-order approximation and a conjugate-gradient solve, making TRPO noticeably slower per update than simpler methods. This is precisely why Proximal Policy Optimization (PPO) was later introduced as a simpler, first-order approximation to the same idea.