Statistical Significance In TCA

A trading desk runs Algo A on 200 orders last month and Algo B on 200 different orders. Algo A averages 3.2 basis points of slippage against arrival price; Algo B averages 2.9 basis points. Someone concludes "Algo B is better, switch everything to it." But slippage per order is noisy — some orders are in illiquid names, some coincide with news, some are just unlucky — and a 0.3 basis-point average gap across 200 orders each with a standard deviation of, say, 8 basis points could easily be nothing but noise. Transaction cost analysis (TCA) needs the same statistical discipline as any other empirical claim: is this difference real, or is it what you'd see by chance even if the two algos were identical?

Think of it like comparing two students' exam averages from a handful of quizzes each. If one scored 82% and the other 80% on five quizzes apiece, nobody would call that proof of a real ability gap — the quiz-to-quiz swings are bigger than the difference itself. You need to know how much the scores bounce around before a gap in averages means anything.

The tool is the standard error of the mean. For $n$ independent orders with slippage standard deviation $\sigma$, the standard error of the average slippage is

$\text{SE} = \frac{\sigma}{\sqrt{n}}$

In words: the uncertainty in your estimate of "typical slippage" shrinks with the square root of how many orders you've observed — four times as many orders only halves the uncertainty. To compare two algos' average slippage, use a two-sample t-statistic,

$t = \frac{\bar{x}_A - \bar{x}_B}{\sqrt{\text{SE}_A^2 + \text{SE}_B^2}}$

In words: the observed gap between the two averages, divided by how much that gap would naturally bounce around if there were no real difference. As a rule of thumb, $|t| > 2$ is roughly the threshold for "this is unlikely to be chance alone" (a two-sided 5% significance level).

Worked example. Algo A: 200 orders, mean slippage 3.2 bps, standard deviation 8.0 bps. $\text{SE}_A = 8.0/\sqrt{200} = 8.0/14.14 = 0.566$ bps. Algo B: 200 orders, mean 2.9 bps, standard deviation 7.5 bps. $\text{SE}_B = 7.5/\sqrt{200} = 0.530$ bps. The gap is $3.2 - 2.9 = 0.3$ bps. The combined standard error is $\sqrt{0.566^2 + 0.530^2} = \sqrt{0.320 + 0.281} = \sqrt{0.601} = 0.775$ bps. So $t = 0.3 / 0.775 = 0.39$ — far below 2. The observed 0.3 bp edge is well within what you'd expect from noise alone with only 200 orders each; there is no statistical basis for preferring Algo B.

Second example, with enough data to matter. Same average gap of 0.3 bps, but now each algo has run on 20,000 orders instead of 200. Standard errors shrink by a factor of $\sqrt{20000/200} = \sqrt{100} = 10$: $\text{SE}_A = 0.0566$ bps, $\text{SE}_B = 0.0530$ bps, combined SE $= 0.0775$ bps. Now $t = 0.3 / 0.0775 = 3.87$ — comfortably significant. The exact same 0.3 basis-point average gap is meaningless at 200 orders and highly significant at 20,000, purely because of sample size. The lesson: a TCA report that quotes an average slippage number without also quoting a sample size and a standard error cannot be evaluated at all.

What this means in practice

TCA reports that rank algorithms, brokers, or venues by average cost alone routinely lead desks to chase noise — switching providers or retuning parameters based on differences that a proper t-test would reveal as statistically indistinguishable from zero. Running the significance test costs nothing beyond tracking the standard deviation and sample count alongside the mean, and it prevents a desk from "optimizing" its execution process into a random walk.

The classic confusion

This is the same Hypothesis Testing and Sample Moments and Their Standard Errors machinery used to evaluate a trading signal, applied to an execution process instead of a return stream — see The Standard Error of the Sharpe Ratio for the closely analogous problem of knowing when a Sharpe ratio is real.