The REINFORCE Algorithm

Q-learning learns values: an estimate of how good each action is in each state, and it acts by picking whichever action currently has the highest estimated value. REINFORCE takes a different route entirely. It skips value estimation and directly learns a policy — a function that outputs a probability of taking each action — and adjusts that policy's parameters using nothing more than one simple, powerful fact: after playing out a full episode and observing its total reward, you can nudge every action you took, in proportion to how good the episode turned out, without ever needing a separate value function at all.

The core trick, in words before symbols

Run the current policy for a full episode — a full trading day, say, taking a sequence of actions and eventually observing a total reward $R$ for the episode. REINFORCE's update rule says: for every action taken along the way, increase the probability of that action if $R$ was good, and decrease it if $R$ was bad, and scale the size of that push by how surprising the action was under the current policy (rarely-taken actions get bigger pushes when they pay off, since the policy has more room to update its belief about them).

$$\nabla_\theta J(\theta) \approx \sum_{t} \nabla_\theta \log \pi_\theta(a_t \mid s_t) \cdot R$$

In words: to improve the policy's parameters $\theta$, for each action $a_t$ taken at each timestep $t$, compute how much increasing that action's log-probability under the current policy $\pi_\theta$ would move the parameters, and scale that direction by the episode's total reward $R$. Summed over the whole episode, this gives a direction to adjust $\theta$: nudge parameters to make good-reward episodes' actions more likely and bad-reward episodes' actions less likely, purely from the reward signal, no separate value estimate required.

Worked sketch: one good episode, one bad one

Suppose a simple policy chooses between "buy" and "hold," and two episodes are played out. Episode 1: the policy chose buy three times in a row, and the total reward was $R = +6$ (good). Episode 2: it chose buy twice and hold once, and the total reward was $R = -4$ (bad). REINFORCE's update, applied to episode 1, pushes the parameters to make "buy" more likely in the states where it was chosen, scaled by $+6$. Applied to episode 2, it pushes the chosen actions to be less likely, scaled by $-4$. Averaged across many such episodes, actions that reliably precede good total rewards get systematically reinforced, and actions preceding bad ones get systematically discouraged — hence the algorithm's name.

REINFORCE's parameter update is, mechanically, one more step of gradient ascent on expected reward, following the same "take a step in the direction that improves things" logic shown above — the difference from ordinary supervised gradient descent is that the "loss" here is an entire episode's total reward, known only after the fact, not a per-example label known upfront.

What this means in practice

REINFORCE's biggest practical drawback is that it uses the entire episode's total reward to scale every single action's update, even actions taken early in the episode that had little to do with a late, decisive outcome — this makes the learning signal extremely noisy (high variance), since one lucky or unlucky late event drags the credit or blame back onto every earlier action indiscriminately. In a multi-step trading strategy, an early entry decision and a late, unrelated market-wide shock both get blamed or credited by the same total $R$, which is exactly the problem actor-critic methods were built to fix by learning a value estimate specifically to reduce this variance.