Cluster-Randomized and Switchback Experiments

Ordinary A/B testing randomizes at the level of individual units, but that isn't always possible: if a new order-routing rule affects an entire trading desk's shared order flow, you can't randomize which of two rules applies to individual orders from that desk without contaminating the comparison, since orders interact with each other in the same market. Cluster-randomized designs solve this by randomizing at the level of whole groups instead — entire desks, entire regions, or entire accounts — each getting one treatment throughout, so interactions stay within a cluster and don't leak across the comparison. Switchback designs handle a related problem: instead of splitting by group, they split by time, randomly assigning the whole system to treatment or control in alternating time blocks (say, alternating hours or days), which is useful when the treatment is a market-wide setting that can't apply to only some participants at once.

Both designs need a matching statistical adjustment: because whole clusters or whole time blocks move together, the number of truly independent observations is far smaller than the number of individual trades or orders, so standard errors computed as if every order were independent will be badly understated and can make noise look like a significant effect.

Switchback experiments carry an extra hazard: effects can carry over from one time block into the next, since a market-wide setting change doesn't necessarily wash out the moment the block ends — inventory built up under one regime, or participants who adapt their behavior mid-experiment, can bleed into the following block's measurement. Careful switchback designs insert a buffer period between blocks, or discard the first portion of each new block, specifically to let such carryover effects fade before data collection resumes.