An 8% versus 11% test on 1000 users each

Question

An A/B test uses 1000 users per arm. Control has 80 conversions ( $8\%$ ); treatment has 110 conversions ( $11\%$ ).

Compute the two-sided p-value for the difference in conversion rates using a two-proportion z-test.

Accepted Answer

Pooled rate under the null. \hat{p} = \frac{80 + 110}{1000 + 1000} = \frac{190}{2000} = 0.095. Standard error. SE = \sqrt{0.095 	imes 0.905 	imes \left(	frac{1}{1000} + 	frac{1}{1000}ight)}. Here 0.095 	imes 0.905 = 0.085975, times 0.002 = 0.00017195, and \sqrt{0.00017195} \approx 0.01311. z and p. The observed difference is 0.11 - 0.08 = 0.03: z = \frac{0.03}{0.01311} \approx 2.29, \qquad p = 2\,[1 - \Phi(2.29)] = 2 	imes 0.0110 \approx 0.022. Interpretation. At p \approx 0.022, the lift is significant at 5% and comfortably clear of the bar, though not at the 1% level. If the two tru…

An 8% versus 11% test on 1000 users each

Your answer

Discussion

Learn the concepts

Related questions