Going from 10% to 5% significance

Question

A two-sided test at \alpha = 0.10 with 80\% power needs 500 users per arm. The team decides to tighten to \alpha = 0.05, keeping the same power and effect. Roughly how many users per arm are needed now?

Accepted Answer

The two constants. At \alpha = 0.10 two-sided, the critical value is z{\alpha/2} = 1.645; at \alpha = 0.05 it is 1.96. - \alpha = 0.10: (1.645 + 0.84)^2 = (2.485)^2 = 6.18. - \alpha = 0.05: (1.96 + 0.84)^2 = 7.84. The ratio. \frac{n{0.05}}{n{0.10}} = \frac{7.84}{6.18} \approx 1.27. Apply it. n{0.05} \approx 1.27 	imes 500 \approx 635 	ext{ per arm}. Halving the false-positive rate from 10% to 5% costs about 27% more data. Takeaway. Tightening from \alpha = 0.10 to 0.05 inflates the sample by about 27%: 500 per arm becomes roughly 635.

Going from 10% to 5% significance

Your answer

Discussion

Learn the concepts

Related questions