Qm
Statistics

Autoplay lifts watch time but hurts a stability guardrail

ab-testingguardrail-metricstrade-offs

A video app tests autoplaying the next clip automatically. The primary metric, watch time per session, is up a significant +9%. But two guardrails move the wrong way: the app-crash rate is up a significant +0.3 percentage points (autoplay strains memory on older phones), and the share of users who turn autoplay off is meaningful.

Should the team ship, and how should they weigh the guardrail regressions?

Your answer

Solving needs a free account

Answers, streaks and solutions unlock when you are signed in. Reading the question and the hint stays free.

Create a free accountSign in

Discussion

Sign in to join the discussion · reading is open to everyone

💡 Discussion rules

  1. No full solutions here. Hints and approaches only.
  2. Complexity, edge cases and intuition are the point.
  3. Interview experiences are welcome. Respect your NDAs.

Loading discussion…

Learn the concepts

The theory behind this question.

A/B TestingCluster-Randomized and Switchback ExperimentsExperimental Design: Blocking and Factorial DesignsMulti-Objective Optimization and Pareto FrontsOnline Experimentation for Model Rollout

Related questions

Clicks up, page speed down: do you ship?More conversions but more returns: ship or not?More email opens, more unsubscribes: what do you do?Revenue per session up, but a retention guardrail dipsWith ten guardrails, how often does one falsely fire?
All questions →