Four strategies, six t-tests, and a hidden error problem

Question

You want to compare the mean returns of four strategies. The instinct is to run all six pairwise t-tests and report the significant ones.

Explain the error-rate problem with all-pairwise t-tests, what one-way ANOVA does instead, and how to follow up properly.

Accepted Answer

The multiplicity problem. Four groups produce \binom{4}{2} = 6 pairwise comparisons. If every strategy truly had the same mean and you tested each pair at 5%, the chance of at least one false positive is about 1 - 0.95^6 \approx 26\% (higher still with more groups). Reporting "the significant pairs" from an all-pairwise sweep therefore surfaces lucky noise a quarter of the time, you'd routinely "discover" a difference among identical strategies. What one-way ANOVA does. Analysis of variance asks a single omnibus question: "are all group means equal, or is at least one different?" It compares …

Four strategies, six t-tests, and a hidden error problem

Your answer

More Statistics questions