The p-value for a conversion-rate A/B test

Question

An A/B test runs 2000 users through each arm. Control gets 100 conversions ( $5.0\%$ ); treatment gets 130 conversions ( $6.5\%$ ).

Compute the two-sided p-value for the difference in conversion rates using a two-proportion z-test.

Accepted Answer

Pool the rate under the null. Under the null of equal true rates, estimate one common rate from both arms combined: \hat{p} = \frac{100 + 130}{2000 + 2000} = \frac{230}{4000} = 0.0575. Standard error of the difference. Using the pooled rate: SE = \sqrt{\hat{p}(1-\hat{p})\left(	frac{1}{n1} + 	frac{1}{n2}ight)} = \sqrt{0.0575 	imes 0.9425 	imes \left(	frac{1}{2000} + 	frac{1}{2000}ight)}. The inside is 0.0575 	imes 0.9425 = 0.05419, times 0.001 = 0.00005419, and \sqrt{0.00005419} \approx 0.00736. The z-statistic and p-value. The observed difference is 0.065 - 0.050 = 0.015, so z = \f…

The p-value for a conversion-rate A/B test

Your answer

Discussion

Learn the concepts

Related questions