Experimental Design: Blocking and Factorial Designs

You want to test whether a new order-execution algorithm reduces trading cost. If you just run it on whichever trades happen to come in, you'll pick up all the natural variation between an illiquid micro-cap trade and a liquid large-cap trade, drowning out the algorithm's actual effect. And if you also want to know whether the algorithm interacts with time-of-day, testing each factor in its own separate experiment doubles your cost and tells you nothing about whether the two factors work together or against each other. Blocking and factorial design are two complementary fixes: blocking strips out a known source of noise before you even look at the treatment effect, and factorial design tests multiple factors in a single run, cheaply revealing how they interact.

An analogy: pairing runners, testing two things at once

To test whether a new shoe makes runners faster, you wouldn't compare a random sprinter wearing the new shoe against a random marathoner wearing the old one — natural speed differences would swamp any shoe effect. Instead you pair up runners of similar ability (a block) and, within each pair, randomly assign one to the new shoe and one to the old — comparing like with like removes the runner-to-runner noise entirely. Separately, if you also want to know whether the shoe works better on wet or dry track, instead of two separate studies you can test shoe type and track condition together in one experiment where every combination appears — a factorial design, which for the same number of runners tells you each factor's effect and whether they interact, at no extra cost over testing one factor alone.

The idea, one symbol at a time

In a blocked design, units are first grouped into blocks $b = 1, \ldots, B$ that are similar on some known nuisance variable (e.g., liquidity tier), and treatment is randomized within each block rather than across the whole sample. The outcome model is $Y_{ib} = \mu + \tau \cdot T_{ib} + \beta_b + \varepsilon_{ib}$, where $\tau$ is the treatment effect, $T_{ib}$ is the treatment indicator, and $\beta_b$ is a block-specific baseline that gets differenced out rather than left in the error term. In plain English: comparing treated and control units within the same block cancels out whatever made that block different, leaving a cleaner estimate of $\tau$ with less noise than an unblocked comparison. In a $2^k$ factorial design with $k$ factors each at 2 levels, every combination of levels is tested, giving $2^k$ cells; this lets you estimate each factor's main effect (its effect averaged over the other factor's levels) and the interaction effect (whether one factor's effect depends on the other factor's level) from the same data.

Worked example 1: blocking by liquidity tier

Suppose 40 trades split into 2 liquidity blocks of 20 each, with the new algorithm randomized within each block. In the high-liquidity block, average cost is 3.0 bps (control) vs 2.6 bps (treated) — a 0.4 bps saving. In the low-liquidity block, average cost is 12.0 bps (control) vs 9.5 bps (treated) — a 2.5 bps saving. An unblocked analysis pooling all 40 trades would have those 0.4 and 2.5 bps effects blurred together with the much larger tier-to-tier baseline gap (3.0 vs 12.0 bps), inflating the estimate's variance. The blocked estimate averages the within-block effects — $(0.4+2.5)/2 = 1.45$ bps — with a standard error reflecting only the tighter within-tier variation.

Worked example 2: a 2x2 factorial by hand

Now test algorithm (new vs old) and time-of-day (open vs midday) together, 25 trades per cell:

	Old algo	New algo
Open	8.0 bps	6.5 bps
Midday	4.0 bps	3.8 bps

Main effect of the new algorithm, averaged over time-of-day: $\frac{(8.0-6.5)+(4.0-3.8)}{2} = \frac{1.5+0.2}{2} = 0.85$ bps saved on average. But the effect isn't constant: it saves 1.5 bps at the open and only 0.2 bps midday — an interaction, meaning the algorithm's benefit depends heavily on when it's used. A single-factor experiment run only at the open would have overstated the algorithm's typical benefit; the factorial design catches this for free.

What this means in practice

Blocking and factorial design are how you get a sharper answer, or several answers at once, from the same experimental budget. In trading contexts, natural blocks include liquidity tier, sector, or volatility regime; natural factorial questions include testing an algorithm change alongside a parameter change to see if they interact. Both techniques rely on treatment still being randomized — within blocks, or within each factorial cell — so the randomization inference logic still applies to the analysis afterward.