New users versus veterans is not a fair diff-in-diff

Question

A product analyst measures a new feature with difference-in-differences. The treated group is a cohort of brand-new users who received the feature; the control is a cohort of long-tenured users who did not. Engagement rose more in the treated cohort, and the analyst reports a positive feature effect.

Explain why comparing new users to veterans breaks parallel trends, and how you would redesign the comparison.

Accepted Answer

New and veteran users are on different natural curves. Brand-new users typically ramp up their engagement over their first weeks as they learn the product and build habits, so their engagement rises steeply on its own. Long-tenured users have already plateaued; their engagement is roughly flat. These are different trajectories baked into where each cohort sits in its life cycle, independent of any feature. Why parallel trends fails. Difference-in-differences assumes both cohorts would have moved together without the feature. But new users would have climbed anyway simply from onboarding, whil…

New users versus veterans is not a fair diff-in-diff

Your answer

More Statistics questions