Equivalence Testing and TOST

An ordinary hypothesis test is built to detect a difference, not to confirm equivalence, "we failed to reject the null of no difference" just as often means the test was underpowered as it means the two things really are the same. If a backtest wants to claim that a faster, cheaper execution algorithm produces essentially the same fill quality as the current one, a standard t-test that fails to find a significant difference is weak evidence at best, since a small or noisy sample would produce that same non-result even if a meaningful difference existed.

Equivalence testing flips the logic: instead of a null hypothesis of "no difference," it sets the null as "the difference is at least as large as some pre-specified margin," and requires you to actively reject that null to conclude equivalence. The standard way to run this is TOST, two one-sided tests: you separately test whether the difference is significantly above the negative margin and significantly below the positive margin; only if both one-sided tests reject do you conclude the true difference falls inside the equivalence margin with the desired confidence.

Comparing two execution algorithms' average slippage, you might pre-specify an equivalence margin of $0.01 per share as "practically indistinguishable." TOST would test whether the observed slippage difference is significantly less than $0.01 in each direction; passing both sides supports genuine equivalence within that margin, whereas an ordinary t-test's non-significant result alone would not.