Actor-Critic Methods

REINFORCE learns a policy directly, which is powerful, but its central weakness is noise: it can only judge whether an action was good or bad by waiting for an entire episode to finish and looking at the total reward, which blames or credits every action in the episode by the same final number, regardless of how much that particular action actually contributed. Actor-critic methods keep REINFORCE's core idea — learn a policy directly, nudge action probabilities based on outcomes — but add a second learned component, the critic, whose only job is to produce a faster, less noisy judgment of how good an action was, without waiting for the whole episode to end.

Two learners, two jobs

An actor-critic system trains two things at once. The actor is the policy — the same thing REINFORCE learns, a function from state to action probabilities. The critic is a value function, estimating how good a given state (or state-action pair) is expected to be going forward, updated the way ordinary value-based methods (like Q-learning) are updated: using the observed reward plus an estimate of what comes next, not waiting for the full episode. The actor's update rule looks almost identical to REINFORCE's, with one crucial substitution: instead of scaling by the raw total episode reward $R$, it scales by the critic's estimate of the advantage — how much better this action was than what the critic expected on average in that state.

$$\nabla_\theta J(\theta) \approx \sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t) \cdot A(s_t, a_t)$$

In words: same update direction as REINFORCE — increase the log-probability of actions that turned out well — but scaled by the advantage $A(s_t, a_t)$, the critic's estimate of "how much better than expected," rather than by the raw, unadjusted total reward from the whole episode. This one substitution is what cuts the noise: the advantage is available step by step, using the critic's running estimate, rather than only at the very end of an episode.

Worked sketch: why advantage beats raw reward

Suppose a trading episode ends with total reward $R = +10$, and one particular action was taken in a state the critic had already learned was typically worth about +9 going forward (a favorable state most of the time, action or no action). REINFORCE would scale that action's update by the full $+10$, treating it as strongly responsible for a large positive outcome. An actor-critic method instead computes the advantage as roughly $10 - 9 = 1$ — the action only added marginally beyond what that state already promised, so its update gets scaled by a small $+1$, not the misleadingly large $+10$. Conversely, an action taken in a state the critic rated as typically worth only +2, in an episode that still ended at $R = +10$ thanks mostly to other actions, gets an advantage closer to $10 - 2 = 8$ — correctly flagged as the action that mattered.

Both the actor and the critic are, mechanically, each running their own version of the gradient-following process shown above — the actor climbing to increase expected advantage-weighted action probability, the critic descending to make its value predictions match observed outcomes more closely. Training two learners simultaneously, each depending on the other's current output, is part of why actor-critic systems are trickier to stabilize than either piece alone.

What this means in practice

Actor-critic methods (and modern variants like PPO and A2C) are the standard choice for training trading agents over long, multi-step sequences — a market-making agent quoting continuously through a session, say — precisely because REINFORCE's episode-level credit assignment gets hopelessly noisy the longer an episode runs. The cost is complexity: two networks to tune instead of one, and instability if the critic's estimates are themselves poor early on, since a bad critic actively misdirects the actor rather than merely failing to help.