Tightening alpha inflates the sample

Question

A two-sided test at \alpha = 0.05 with 80\% power needs 800 users per arm. The team now wants to run at a stricter \alpha = 0.01, keeping the same power and the same effect. Roughly how many users per arm are needed now?

Accepted Answer

Where the sample size comes from. For fixed effect and power, the required sample scales with (z{\alpha/2} + z\beta)^2. Tightening \alpha raises the critical value z{\alpha/2}, which raises n. The two constants. - \alpha = 0.05: z{\alpha/2} = 1.96, so (1.96 + 0.84)^2 = 7.84. - \alpha = 0.01: z{\alpha/2} = 2.576, so (2.576 + 0.84)^2 = (3.416)^2 = 11.67. The ratio. \frac{n{0.01}}{n{0.05}} = \frac{11.67}{7.84} \approx 1.49. Apply it. n{0.01} \approx 1.49 	imes 800 \approx 1{,}190 	ext{ per arm}. So demanding a false-positive rate five times smaller costs roughly 50% more data, not five times m…

Tightening alpha inflates the sample

Your answer

Discussion

Learn the concepts

Related questions