Tukey's HSD and Post-Hoc Comparisons

An ANOVA test across several groups only tells you whether at least one group's mean differs from the others — it doesn't say which ones. The obvious next step, running a t-test on every pair of groups, inflates the chance of a false positive: with five groups there are ten pairwise comparisons, and even at a 5% significance level per test, the probability that at least one comes back "significant" purely by chance climbs well above 5%. Tukey's HSD ("honestly significant difference") test is built specifically to run all pairwise comparisons while keeping the overall, family-wise false-positive rate at the stated level.

It does this by using the distribution of the studentized range — the spread between the largest and smallest sample mean, scaled by the pooled standard error — rather than treating each pairwise t-test as independent. Any two group means are declared significantly different only if their gap exceeds a critical value derived from this studentized-range distribution, which is wider than an ordinary t-test's critical value precisely to account for testing multiple pairs at once.

Comparing average returns across five trading strategies, an ANOVA might reject the hypothesis that all five have equal means. Running Tukey's HSD afterward could then show that strategies A and C differ significantly from each other, while none of the other eight pairwise gaps clear the (wider) HSD threshold — pinpointing exactly which pair drives the overall ANOVA result without the inflated false-positive rate that ten separate uncorrected t-tests would carry.