The distribution of the smallest p-value

Question

You run k independent tests, all under the null, and report only the smallest p-value P{(1)} = \min(P1, \dots, Pk). Find the distribution of P{(1)} and its expected value. Explain why comparing it to 0.05 as if it were a single test is dangerously misleading.

Accepted Answer

Distribution of the minimum. Under the null each Pi \sim 	ext{Uniform}(0,1) independently, so the minimum exceeds u only if all k do: \mathbb{P}(P{(1)} u) = \prod{i=1}^{k}\mathbb{P}(Pi u) = (1 - u)^k. Hence the CDF is \mathbb{P}(P{(1)} \le u) = 1 - (1 - u)^k, which is the Beta(1, k) distribution, sharply concentrated near 0. Expected value. For Beta(1,k), E[P{(1)}] = \frac{1}{k + 1}. With k = 20 tests, the smallest p-value averages 	frac{1}{21} \approx 0.048, right around the "significant" threshold, purely by chance and with nothing real going on. Why comparing to 0.05 is misleading. A sin…

The distribution of the smallest p-value

Your answer

More Statistics questions