Explore Versus Exploit

The problem: you're choosing between several slot machines (or trading signals, or market-making strategies) with unknown win rates, and you get to pull one lever at a time. Every pull teaches you a little about that lever's true win rate, but every pull spent testing an unproven lever is a pull not spent on the lever you currently believe is best. Pull only your current favorite forever and you might be stuck on a mediocre option because you never gathered enough evidence about a better one. Spend forever testing every option equally and you waste pulls on options you already know are bad. This tension — explore to learn, or exploit what you already know — is one of the cleanest decision-theory questions asked in interviews.

Two basic strategies

Epsilon-greedy is the simplest fix: with small probability $\varepsilon$ (say 10%), pull a random lever to explore; the rest of the time, pull whichever lever currently has the best estimated win rate. It's easy to reason about and easy to implement, but it explores forever at a constant rate even after you're quite confident which lever is best, wasting pulls on options you've already ruled out well.

Upper confidence bound (UCB) strategies do better by making exploration adaptive: pull the lever that maximizes estimated win rate plus a bonus for uncertainty — levers pulled fewer times get a bigger bonus, so they naturally get explored more early on, and as evidence accumulates the bonus shrinks and the algorithm settles onto genuinely pulling its best-estimated lever almost exclusively. This is the shape of "optimism in the face of uncertainty": assume an under-tested option might be great until the evidence says otherwise.

Worked example: two arms, ten pulls

Two levers, A and B, true (unknown to you) win probabilities 0.6 and 0.4. Suppose your first four pulls happened to be on A, giving 3 wins out of 4 (75% observed), and B is untested (no data, so treated optimistically). Under epsilon-greedy with $\varepsilon = 0.2$: 80% of the time pull A (the current best), 20% of the time pull a random lever. Over the next 6 pulls, in expectation about 1.2 of them go to B purely by the exploration probability, regardless of how confident you are about A — the exploration rate doesn't taper.

Under a UCB-style rule, B's complete lack of data gives it a large uncertainty bonus, so the algorithm would likely pull B next regardless of A's promising 75% observed rate, specifically because B is unproven. Suppose that exploratory pull of B wins — now B has 1 win from 1 pull (100% observed, but on very little data), and the uncertainty bonus for B shrinks a bit while still being sizable. The algorithm keeps sampling B a few more times before the bonus shrinks enough for A's larger, more reliable sample of wins to dominate the decision. This adaptivity — explore hard when data is thin, taper off automatically as data accumulates — is exactly what a constant $\varepsilon$ can't do.

What this means in practice

This tradeoff appears any time you're allocating limited trials across options with unknown payoffs and updating beliefs as you go — testing new trading signals against a proven one, allocating capital across strategies with uncertain live performance, or A/B-testing execution algorithms. The interview-relevant takeaway is that "always do what looks best right now" is a systematically bad long-run policy whenever your current estimates are based on limited data, and the fix isn't to explore randomly forever but to explore adaptively — more when uncertain, less as evidence accumulates.