A feature's lift shrinks every week, what is going on?

Question

A social app ships a redesigned home feed to a random half of users and measures time spent per day. The treatment shows +8% in week 1, +5% in week 2, +2% in week 3, and roughly 0% by week 4.

What effect is most likely driving this pattern, and how would you confirm it?

Accepted Answer

Name it: the novelty effect. When something looks new, users click around and explore it simply because it is unfamiliar, not because it is genuinely better. That curiosity inflates engagement right after launch. As the newness wears off, behavior drifts back toward normal, so the measured lift decays week over week, exactly the 8, 5, 2, 0 pattern here. If you had stopped the test in week 1 and shipped on +8%, you would have badly overstated the real, long-run impact. How to confirm it. 1. Run the test longer. If the lift keeps decaying toward zero and flattens there, that is the signature of…

A feature's lift shrinks every week, what is going on?

Your answer

More Statistics questions