The Policy Gradient Theorem

Value-based methods like Q-Learning work by scoring every action and then taking the best one. That is fine when there are five actions. It falls apart when the action is a number — how many shares to send, how wide to quote, what fraction of the book to lift — because you cannot enumerate a continuum, and it falls apart again when the right behaviour is deliberately random, as it is for any market maker who does not want to be predictable. What you want instead is to hold the decision rule itself in your hands, with dials on it, and turn the dials in whichever direction makes money.

The obstacle is that you cannot differentiate profit with respect to those dials. Profit comes out of the market, and the market is not a function you have source code for. You cannot compute "how would my P&L have changed if I had quoted one tick tighter," because you never ran that world.

Here is the way around it, and it is worth sitting with before any symbols appear. Imagine a coach who does not understand the sport. She cannot analyse why a play works. All she can do is call plays from a deck, watch the score, and then reshuffle the deck: plays that were followed by good outcomes get more copies, plays followed by bad outcomes get fewer. She never learns the physics of the game, yet the deck steadily improves. The policy gradient theorem is the precise, unbiased version of that reshuffling — and crucially, it needs only the probabilities she assigned to her own plays, never a model of the game.

Setting up the dials

Write the policy as $\pi_\theta(a \mid s)$: the probability of choosing action $a$ when the state is $s$, where $\theta$ is the vector of parameters — the dials, or the weights of a neural network. Unlike a value table, this outputs probabilities directly, so randomness is built in rather than bolted on.

The thing we want to maximise is $J(\theta)$, the expected total reward earned over an episode when actions are drawn from $\pi_\theta$. In words: the average P&L of the strategy, if you ran it many times. We would like $\nabla_\theta J(\theta)$, the direction in dial-space that raises that average fastest.

The theorem

$$\nabla_\theta J(\theta) = \mathbb{E}\!\left[ \nabla_\theta \log \pi_\theta(a \mid s) \; Q^{\pi}(s,a) \right]$$

In plain English: to improve the policy, take each action you actually took, ask "which way should I move the dials to make this action more likely," and scale that direction by how good the action turned out to be. Good actions get pushed up hard, mediocre actions get a gentle push, bad actions get pushed down.

Two things are remarkable about this. First, $\nabla_\theta \log \pi_\theta(a \mid s)$ — called the score function — depends only on your own policy. You wrote it; you can differentiate it. The environment never appears inside a derivative. Second, the expectation is over trajectories you can simply sample, so an average over a batch of real episodes is an unbiased estimate of the true gradient. No model, no simulator, no dynamics.

Worked example: one honest update

Take the simplest possible policy: two actions, two dials $\theta = (\theta_1, \theta_2)$, probabilities from a softmax. Start at $\theta = (0,\,0)$, so both actions are equally likely at $0.5$.

For a softmax policy the score function has a famously clean form: for the action you took, the derivative is $1 - \pi(a)$; for every action you did not take, it is $-\pi(a)$. Read it as "raise the one you played, lower the rest, in proportion to how surprised you should have been."

Now sample. The agent plays action 1 and collects a return of $R = 1$.

Step 1 — the score function at the sampled action:

$$\nabla_{\theta_1} \log \pi = 1 - 0.5 = 0.5, \qquad \nabla_{\theta_2} \log \pi = 0 - 0.5 = -0.5 .$$

Step 2 — scale by the return:

$$\hat{g} = R \times (0.5,\ -0.5) = (0.5,\ -0.5) .$$

Step 3 — take a gradient ascent step with learning rate $\alpha = 0.4$:

$$\theta \leftarrow (0,\,0) + 0.4 \times (0.5,\ -0.5) = (0.2,\ -0.2) .$$

Step 4 — read off the new probabilities. With $e^{0.2} = 1.221$ and $e^{-0.2} = 0.819$ summing to $2.040$:

$$\pi(1) = \frac{1.221}{2.040} = 0.599, \qquad \pi(2) = \frac{0.819}{2.040} = 0.401 .$$

The probability of the action that worked rose from $50\%$ to about $60\%$. Nothing about the environment was ever consulted.

Worked example: why a baseline is nearly free money

Now suppose both actions pay well: action 1 returns 10, action 2 returns 12. Start again at $\theta = (0,0)$ with both probabilities $0.5$, and sample two episodes — one of each action.

Without a baseline:

$$\hat{g}_1 = 10 \times (0.5,\ -0.5) = (5,\ -5), \qquad \hat{g}_2 = 12 \times (-0.5,\ 0.5) = (-6,\ 6)$$

Averaging them gives $(-0.5,\ 0.5)$. The direction is right — favour action 2 — but look at the scale. The signal is $0.5$ while the individual samples were $\pm 5$ and $\pm 6$. You are extracting a whisper from two shouts that nearly cancel, and with real noise you would need thousands of episodes to hear it.

With a baseline $b = 11$, the average return, subtracted from each return:

$$\hat{g}_1 = (10-11) \times (0.5,\ -0.5) = (-0.5,\ 0.5), \qquad \hat{g}_2 = (12-11) \times (-0.5,\ 0.5) = (-0.5,\ 0.5)$$

The average is $(-0.5,\ 0.5)$ — identical. But now the two samples agree perfectly instead of disagreeing violently. Same answer, a fraction of the variance.

That is the single most important practical corollary of the theorem: you may subtract any quantity that does not depend on the action without biasing the gradient, because the probabilities sum to one and the extra term cancels in expectation. Choosing $b = V(s)$, the state's average value, turns $Q(s,a) - V(s)$ into the advantage — "was this action better or worse than typical here" — and every modern algorithm from Actor-Critic Methods to Proximal Policy Optimization is built on it.

Policy gradient is still gradient ascent, so it inherits every pathology of a step-size choice. Drag the learning rate below and watch the same three failure modes you get anywhere else — crawling, converging, and overshooting into instability.

What this means in practice

If you are training an execution agent, the policy gradient is what lets the action be "send $x\%$ of remaining size" rather than a menu of five choices. If you are training a market maker, it is what lets the quote be genuinely stochastic so it is not picked off by pattern-matching counterparties.

What breaks in practice is almost always variance, not bias. Financial rewards are dominated by noise — the same policy on the same day can make or lose money for reasons it did not cause. So the gradient estimate is honest but extremely wobbly, and vanilla REINFORCE on market data typically needs far more episodes than you have. The fixes are all variance reductions: subtract a learned baseline, use advantage estimates over short windows, average over large batches, and clip how far the policy can move in a single update.

The theorem is the bridge from "score the actions" to "tune the behaviour", and it underwrites The REINFORCE Algorithm, Actor-Critic Methods and Generalized Advantage Estimation alike.