Why randomize? Website redesign and signups

Question

Your team shipped a website redesign and signups rose the following month, so leadership credits the redesign. But that same month included a paid marketing push and a seasonal traffic bump.

Explain why the before-after jump cannot establish that the redesign caused more signups, and design a test that can. Why is randomization the key ingredient?

Accepted Answer

Why the before-after jump is not enough. The redesign did not launch in isolation. The marketing push and seasonality also changed between the two periods, and either could explain the higher signups on its own. These are confounders that move together with the redesign in time, so a simple before-versus-after comparison cannot separate the redesign's effect from everything else that changed. Time-based comparisons are wide open to anything else happening at the same time. A test that can prove causation. Run a randomized A/B test concurrently: randomly assign each visitor to see the old site…

Why randomize? Website redesign and signups

Your answer

More Statistics questions