Monte Carlo Control

Imagine training a trading agent by having it try a strategy for an entire day, tallying up the total profit at the end, and only then asking "was that a good idea?" — never peeking at intermediate results. That's the core idea of Monte Carlo control: play a full episode (a complete trading day, a full game, a finished backtest run) to its end, record the total reward that resulted, and use that to judge how good each action taken along the way actually was, one full episode at a time.

Concretely, the method estimates the value $Q(s,a)$ of taking action $a$ in state $s$ by averaging the total return that followed every time that state-action pair was actually visited across many episodes — no formula for the environment's dynamics is needed, only the observed outcomes. After enough episodes, the policy is improved by picking, at each state, whichever action currently has the highest estimated $Q(s,a)$, then more episodes are run under the improved policy, and the cycle repeats.

The tradeoff versus other reinforcement learning methods is patience versus bias: because Monte Carlo control only updates estimates after a full episode ends, it needs episodes that actually terminate and can be slow and high-variance (one lucky or unlucky episode swings the estimate), but unlike temporal-difference methods it never bootstraps off a possibly-wrong estimate of a future state, so its estimates stay unbiased.