Proximal Policy Optimization

Imagine retraining a junior trader after one lucky week. The temptation is to overhaul their entire playbook to match whatever they happened to do that made money. That is dangerous: one good week is noisy evidence, and a full rewrite could destroy parts that were working for unrelated reasons. The sane approach is to nudge behaviour a little toward the profitable pattern, check how it performs, then nudge again. Proximal Policy Optimization, PPO, is that caution turned into an algorithm for training a reinforcement-learning agent.

An RL agent has a policy — a rule mapping situations to actions, written $\pi_\theta$, where $\theta$ is the adjustable weights that define it. Training adjusts $\theta$ so that action sequences that earned more reward become more likely. Policy gradient methods, which move $\theta$ directly uphill on reward, can take a step so large that the policy collapses — it starts acting on situations it never experienced, generating garbage data that makes the next update worse. PPO's entire contribution is a governor on step size.

The mechanism: a leash, not a compass

Standard policy gradient asks "which direction increases expected reward?" and steps that way. PPO asks the same question but also asks "how far did the new policy drift from the old one on this decision?" and refuses to reward drifting too far, even if the reward signal says to.

Define the probability ratio for a given action $a$ in situation $s$:

$$r(\theta) = \frac{\pi_\theta(a \mid s)}{\pi_{\theta_{\text{old}}}(a \mid s)}$$

In words: how much more (or less) likely the new policy makes this action compared to the policy you started the update from. If $r = 1.4$, the new policy is 40% more inclined toward that action; if $r = 0.6$, it is 40% less inclined.

Multiply that ratio by $\hat{A}$, the advantage — a number estimating how much better this action was than the policy's average action in that situation (positive means better than average, negative means worse). The raw policy-gradient objective is $r(\theta)\hat{A}$: push $r$ up when the advantage is positive, down when it's negative. PPO's fix is to clip that ratio:

$$L^{\text{CLIP}}(\theta) = \min\Big(r(\theta)\hat{A},\ \text{clip}(r(\theta),\, 1-\epsilon,\, 1+\epsilon)\hat{A}\Big)$$

In plain English: compute the reward you'd get from moving the ratio freely, and separately compute the reward from a version of the ratio capped to stay within $\pm\epsilon$ (typically $\epsilon = 0.2$, so the ratio is clamped between 0.8 and 1.2). Take whichever of the two is worse (the minimum). That last step is the trick — it means the objective never rewards the policy for straying outside the leash, even when straying would look profitable in this update.

Watch the step-size control: too large a step overshoots and oscillates, too small crawls. PPO's clip is exactly a mechanism for keeping every single update inside the "converging" zone instead of the "overshooting" one, applied specifically to how far the policy's behaviour is allowed to move, not to the raw learning rate.

Worked example 1: clipping a good action

Suppose the old policy assigned probability $0.30$ to an action, and a gradient step would push it to $0.50$, giving $r = 0.50/0.30 = 1.67$, with advantage $\hat{A}=+2$ (it worked out well). Unclipped objective: $r\hat{A}=1.67\times2=3.33$. Clipped ratio caps at $1.2$, giving $1.2\times2=2.4$. PPO takes the minimum: $2.4$. Past $r=1.2$, the gradient of this term is zero — the optimizer gets no further reward for pushing this action's probability up beyond the leash, even though the raw objective would say to.

Worked example 2: clipping a bad action from over-correcting

Now an action has advantage $\hat{A}=-1.5$ and a step would drop its ratio to $r=0.55$. Unclipped: $0.55\times(-1.5)=-0.825$. Clipped ratio floors at $0.8$, giving $0.8\times(-1.5)=-1.2$. The minimum of the two is $-1.2$, so PPO uses the clipped, more negative value — it does not soften the penalty here. Clipping only ever blocks reward for overshooting past the trusted region; it never softens a deserved penalty. That asymmetry is what keeps the policy from swinging wildly in either direction.

What this means in practice

PPO is the default choice for training RL agents — including reward-model-tuned language models — because it is far more forgiving of hyperparameter choices than earlier policy-gradient methods, while avoiding the collapse that plain policy gradient suffers on noisy reward signals, which describes almost every real trading environment. In a market-making or execution agent, the advantage estimate $\hat{A}$ is itself noisy, so the clip stops one lucky or unlucky sequence of fills from rewriting the whole policy in one update.