A "surprise discount" banner whose magic wears off

Question

A food-delivery app tests a playful "surprise discount" banner on the home screen with a random half of users and measures orders per user. Orders are up 6% in week 1, 4% in week 2, 2% in week 3, and back to flat by week 4.

What is the most likely explanation, and how would you confirm it before deciding to keep the banner?

Accepted Answer

The novelty effect. A new, eye-catching banner prompts people to tap it and place an order just to see what the surprise is. That behavior is powered by curiosity about something new, not by a lasting change in how much people want to order. Once the banner is familiar, order rates drift back to normal, so the lift decays from 6% toward zero, the pattern shown. How to confirm it. 1. Let the test run longer and watch the lift flatten; if it settles at zero, the early gain was transient. 2. Compare new users (who never saw the app without the banner) against longtime users; a spike concentrated…

A "surprise discount" banner whose magic wears off

Your answer

More Statistics questions