Contextual Bandits

A plain multi-armed bandit is blind: it tracks one payout estimate per arm and assumes that estimate is the whole story, regardless of anything else happening in the world. That is a poor fit for most real trading decisions, where a strategy's expected payoff genuinely depends on current conditions — a momentum signal that works well in a trending, low-volatility regime can lose money in a choppy, high-volatility one. A contextual bandit fixes this by conditioning the decision on a context: a feature vector describing the current situation (recent volatility, time of day, macro regime indicators), and learning a separate, context-dependent estimate of each arm's payoff instead of one flat number per arm.

The added piece: context before the choice

The mechanics are the same as a plain bandit — pick an arm, observe a payout, update your estimates — with one addition at the front: before choosing, you observe a context vector $x_t$ describing the current round. Each arm's estimated payoff is now a function of that context, $\hat\mu_i(x_t)$, rather than a single fixed number, typically fit with a regression model (linear regression is the simplest, common choice) mapping context to expected payoff, one such model per arm. The exploration mechanics from plain bandits (Thompson sampling, upper-confidence-bound reasoning) carry over unchanged, just applied to the context-conditioned estimate rather than a flat average.

Worked example: volatility-conditioned signal choice

Suppose two strategies are the arms: A is a mean-reversion signal, B is a momentum signal, and the context is a single feature, realized 20-day volatility. A plain bandit, ignoring volatility entirely, would track one lifetime average for A and one for B and always favor whichever has the higher overall average — suppose that history shows A averaging +0.15%/day and B averaging +0.10%/day, so a plain bandit locks onto A permanently. A contextual bandit instead fits $\hat\mu_A(\text{vol})$ and $\hat\mu_B(\text{vol})$ separately, and might discover that A's edge is concentrated entirely in low-volatility periods (A returns +0.35% when vol is below 12%, but -0.10% when vol is above 20%), while B is the reverse (B returns +0.02% in low vol, +0.25% in high vol). The contextual bandit, observing today's volatility reading, correctly switches to B on high-volatility days — something the plain bandit's single flat average could never express, because it never separated "A's average" into the regimes where it actually holds.

Fitting $\hat\mu_i(x)$ per arm is literally a regression problem, one instance per arm, of exactly the kind shown above — context on the x-axis, expected payoff on the y-axis, fit with visible residuals. A contextual bandit is, mechanically, running one such regression per arm and picking whichever arm's fitted line (plus its exploration bonus) is highest for today's context.

What this means in practice

Contextual bandits are the right tool whenever "which strategy to run" genuinely depends on observable, available-before-you-decide conditions — regime indicators, recent realized volatility, time-of-day liquidity patterns — and a plain bandit's single flat ranking would be actively misleading. The context has to be genuinely available before the decision and before the payout is known; using a feature that secretly leaks information about the outcome (a common bug) silently inflates apparent performance in backtests.