Did the new checkout button actually lift conversions?
Asked at Amazon
An e-commerce A/B test runs 4,000 visitors through the old checkout (control) and 4,000 through a redesign (treatment). Control converts 200 visitors (5.0%); treatment converts 250 (6.25%). The product team is thrilled by the "25% relative lift."
Test whether the difference in conversion rates is statistically significant using a two-proportion z-test, and comment on the "25% lift" framing.
Your answer
This one is open-ended. Work it through, then check your reasoning against the full solution.