Equivalence Testing and TOST

An ordinary hypothesis test is built to detect a difference, not to confirm equivalence — "we failed to reject the null of no difference" just as often means the test was underpowered as it means the two things really are the same. If a backtest wants to claim that a faster, cheaper execution algorithm produces essentially the same fill quality as the current one, a standard t-test that fails to find a significant difference is weak evidence at best, since a small or noisy sample would produce that same non-result even if a meaningful difference existed.

Equivalence testing flips the logic: instead of a null hypothesis of "no difference," it sets the null as "the difference is at least as large as some pre-specified margin," and requires you to actively reject that null to conclude equivalence. The standard way to run this is TOST — two one-sided tests: you separately test whether the difference is significantly above the negative margin and significantly below the positive margin; only if both one-sided tests reject do you conclude the true difference falls inside the equivalence margin with the desired confidence.

Comparing two execution algorithms' average slippage, you might pre-specify an equivalence margin of $0.01 per share as "practically indistinguishable." TOST would test whether the observed slippage difference is significantly less than $0.01 in each direction; passing both sides supports genuine equivalence within that margin, whereas an ordinary t-test's non-significant result alone would not.