When the "pooled" t-test quietly lies to you

Question

You compare a mean between two groups: a small, tight group ( $n_1 = 10$ , standard deviation $\approx 2$ ) and a large, volatile group ( $n_2 = 100$ , standard deviation $\approx 10$ ). A junior analyst runs the standard pooled-variance two-sample t-test, which assumes both groups share one common variance.

Explain why the pooled test is the wrong tool here, what Welch's t-test does differently, and why Welch should arguably be your default.

Accepted Answer

What the pooled test assumes. The classic two-sample t-test estimates a single pooled variance sp^2 = \frac{(n1 - 1)s1^2 + (n2 - 1)s2^2}{n1 + n2 - 2}, and uses it for both groups. This is only valid when the two groups truly have equal variance. Here s1^2 = 4 and s2^2 = 100 are wildly different, and the larger group's degrees of freedom dominate the pool. The small, tight group gets assigned a variance far bigger than its own, distorting its standard error and the resulting test. Why the damage is worst with unequal n and unequal variance. When the smaller group has the larger variance, the p…

When the "pooled" t-test quietly lies to you

Your answer

More Statistics questions