Counterfactual Evaluation of Execution Policies

A desk has a new, learned execution policy and wants to know whether it would actually perform better than the current one — but nobody wants to run an untested policy live with real money as the first test. The historical logs, however, only record what happened under the old policy: which venues it routed to, when it canceled and reposted, what rate it chose. Counterfactual evaluation is the set of techniques for estimating how a different policy would have performed, using only that old-policy data, without ever deploying the new policy for real.

Why you can't just replay the logs

The naive approach is to look through the historical log for moments where the old policy happened to take the same action the new policy would have taken, and average the outcomes over just those moments. This works, but only if the new policy overlaps enough with the old one's actual behavior — if the new policy wants to route far more often to a venue the old policy rarely used, there's almost no historical data to learn from at exactly the states where they disagree, and the estimate becomes noisy or simply unavailable. This is the same overlap problem seen throughout causal inference: you can't estimate the effect of a treatment nobody in your data ever received.

Importance weighting as the standard fix

The workhorse technique, inverse propensity weighting, reweights the historical outcomes rather than discarding disagreeing actions outright. For each logged decision, compute the ratio of how likely the new policy would have been to take that same action versus how likely the old (logging) policy actually was to take it:

$$w = \frac{\pi_{\text{new}}(a \mid s)}{\pi_{\text{old}}(a \mid s)} .$$

In plain English: if the new policy would have taken this exact action much more often than the old policy did, that observation gets upweighted (it's rare evidence, so it needs to count for more); if the new policy would rarely have taken it, the observation gets downweighted. Averaging the logged outcome times this weight across all historical decisions gives an estimate of the new policy's expected performance — unbiased under reasonable conditions, but with variance that explodes when the two policies disagree substantially, which is exactly the overlap problem showing up again in the estimator's noise rather than its bias.

Worked example: reweighting a routing decision

The old routing policy sent 80% of small orders in a given context to venue A and 20% to venue B, with average realized cost of 3bp on A and 5bp on B in the log. A new policy would send only 20% to A and 80% to B in that same context. Applying the weight to each logged fill: A-fills get weight $0.20/0.80 = 0.25$, B-fills get weight $0.80/0.20 = 4.0$. The weighted average cost estimate for the new policy becomes:

$$\frac{0.25 \times 3 \times 0.80 + 4.0 \times 5 \times 0.20}{0.25 \times 0.80 + 4.0 \times 0.20} = \frac{0.6 + 4.0}{0.2+0.8} = 4.6 \text{ bp} .$$

The estimate leans heavily on the small 20% slice of historical B-fills, upweighted 4×, which is exactly why this estimate carries much more uncertainty than the old policy's own logged average — there simply isn't as much data at the states the new policy prefers.

What this means in practice

Counterfactual evaluation is the standard way to screen a new execution policy before risking any live capital on it, but its estimates should always be reported with a confidence interval — a point estimate alone hides how much of it rests on thin, heavily-reweighted data. When a proposed policy differs too radically from the logging policy for any reasonable overlap to exist, the honest answer is that offline data cannot evaluate it, and a small, carefully monitored live pilot is the only remaining option.