A one-sided A/B test that must prove improvement

Question

A team will only ship the treatment if it is a genuine improvement, so they pre-register a one-sided test (treatment rate $>$ control rate). Each arm has 800 users: control converts at $5\%$ (40 users), treatment at $7.5\%$ (60 users).

Compute the one-sided p-value for the improvement using a two-proportion z-test.

Accepted Answer

Pooled rate under the null. \hat{p} = \frac{40 + 60}{800 + 800} = \frac{100}{1600} = 0.0625. Standard error. SE = \sqrt{0.0625 	imes 0.9375 	imes \left(	frac{1}{800} + 	frac{1}{800}ight)}. Here 0.0625 	imes 0.9375 = 0.05859, times 	frac{2}{800} = 0.0025, giving 0.00014648, and \sqrt{0.00014648} \approx 0.012103. z and one-sided p. The observed difference is 0.075 - 0.050 = 0.025: z = \frac{0.025}{0.012103} \approx 2.07, \qquad p{	ext{one}} = 1 - \Phi(2.07) \approx 0.019. Why one-sided here. Because the team pre-committed to only caring about improvement, they use the single upper tai…

A one-sided A/B test that must prove improvement

Your answer

Discussion

Learn the concepts

Related questions