Online Experimentation for Model Rollout

A canary release checks that a new model doesn't obviously break anything. Online experimentation goes further: it asks whether the new model is actually better, with enough statistical rigor to trust the answer. This means randomly assigning traffic to the candidate and the incumbent, deciding in advance which metric and threshold count as success, and running a proper statistical test rather than just watching two lines on a dashboard and deciding by eye which looks nicer.

Designing the experiment before running it

Four decisions have to be made before the experiment starts, not after seeing results: the randomization unit (per-order, per-account, per-day — whichever avoids one model's decision contaminating what the other sees), the primary metric (realized P&L per trade, fill quality, a proxy loss — picked because it's the thing that actually matters, not just the easiest thing to log), the minimum detectable effect (how large a difference is worth caring about, which determines how much traffic and time the test needs), and the stopping rule (a fixed sample size or duration decided up front — repeatedly peeking at results and stopping the moment they look good inflates the false-positive rate, a well-known trap called "peeking").

Worked example

A desk wants to know if a new execution-timing model reduces slippage versus the current one. They set the primary metric as slippage in basis points per order, decide a 2 basis point improvement is the smallest difference worth caring about, and using a power calculation determine they need roughly 4,000 orders per arm to detect that effect reliably. Orders are randomly assigned per-order to candidate or incumbent, run for eleven trading days until the sample size is hit, and only then is the test unblinded: the candidate averages 14.1 basis points of slippage against the incumbent's 16.3, a 2.2 basis point improvement, and a two-sample t-test gives $p = 0.02$ — below their pre-registered $0.05$ threshold. Because the sample size, metric, and threshold were all fixed beforehand, the desk can trust the result rather than wondering whether they'd have kept running the test longer if the first week had looked worse.

What this means in practice

Online experiments for model rollout share the same discipline as any A/B test, plus one financial wrinkle: market conditions during the test window are themselves a random draw, so a candidate that wins during a calm week isn't guaranteed to hold up in a volatile one — running the test across varied conditions, or explicitly stratifying by regime, guards against a lucky-window result being mistaken for a genuine improvement.