How to actually prove two things are the same

Question

A team wants to migrate from an expensive data vendor to a cheaper one, but only if the cheaper feed is practically equivalent. They run a t-test, find no significant difference, and conclude "the feeds are the same."

Explain why a non-significant t-test cannot establish equivalence, and how the two one-sided tests (TOST) procedure does it properly.

Accepted Answer

The logical gap. A standard t-test's null is "no difference." Failing to reject it means you couldn't find a difference, which is equally consistent with "there's no difference" and with "the test was too weak to see one." So "not significant" never proves sameness; with a tiny sample you'd trivially "prove equivalence" by being uninformative. You cannot confirm the null by failing to reject it. Define what "equivalent" means first. Equivalence is not "difference exactly zero" (impossible to prove and rarely what you need). Pick an equivalence margin \delta: the largest difference that would …

How to actually prove two things are the same

Your answer

More Statistics questions