Exploration vs Exploitation

A trader has found a strategy that reliably makes a modest, steady profit. Every day brings a choice: run the known strategy again (exploit what already works), or spend some capital testing a new, untested variant that might be much better, or might be worse, or might simply lose money while you find out (explore). This tension — squeeze the thing you already know works, or spend resources learning whether something else works better — is the exploration-exploitation trade-off, and it sits underneath every reinforcement-learning algorithm, not just the exotic ones.

Why you can't just do both

The trade-off is real because exploring costs something concrete: capital risked on an untested idea, time spent not running the known-good strategy, or in a physical experiment, patients given an unproven treatment instead of the current best one. If exploring were free, you would simply test everything and then exploit the best option forever — no dilemma. The dilemma exists specifically because information has a price, and pure exploitation risks getting permanently stuck on a merely-good option while a better one goes forever untested, while pure exploration wastes resources on options you already have enough evidence to rule out.

A worked example: two slot machines

Suppose two trading signals, A and B, are each tested for ten days. Signal A has produced returns averaging +0.4% per day with low variance across the ten tests — you have a fairly confident estimate of its quality. Signal B has only been tried twice, once at +2.0% and once at -1.5%, averaging +0.25% but on far less data. Pure exploitation looks only at the averages, sees A ahead (0.4% versus 0.25%), and runs A forever. But B's average is built on two observations with wildly different outcomes — the true long-run average of B could plausibly be well above A's, or well below it; you simply don't have enough evidence yet to know. A sensible exploration policy accounts for this uncertainty directly: it does not merely rank by the sample average, it gives extra weight to options that have been tried few times, precisely because their estimated average is untrustworthy, and might allocate a small amount of ongoing capital to B specifically to narrow that uncertainty down, even though A currently looks better on paper.

A running average like the one above illustrates why: an option's sample average is noisy and can swing wildly with only a handful of observations, settling down only as more trials accumulate. Judging which option is truly better based on a handful of early observations, the way pure exploitation does, means judging based on the noisiest part of that curve.

What this means in practice

Every method covered under reinforcement learning — multi-armed bandits, Thompson sampling, contextual bandits, and the deeper agent-based methods like Q-learning — is, underneath the specific mechanics, a different concrete answer to the same question: given your current uncertainty about each option's true value, how much should you explore versus exploit right now? A firm allocating capital across several new, unproven strategies faces exactly this problem: allocate too conservatively and a genuinely superior new strategy never gets enough capital to prove itself; allocate too aggressively into untested ideas and you bleed capital on strategies that a slightly more patient evaluation would have ruled out.