One metric to decide, several to keep you honest

Question

A test increases click-through by making the page auto-refresh aggressively. Clicks are up, the team calls it a win. But nobody checked page-load time, unsubscribe rate, or long-term retention.

Explain the roles of an overall evaluation criterion (OEC) and guardrail metrics, and how they prevent this kind of hollow victory.

Accepted Answer

The failure mode. Optimizing a single narrow metric invites gaming: aggressive auto-refresh mechanically inflates clicks without creating any real value, and may actively harm the experience (slower pages, annoyed users who churn). A metric that's easy to move in a way that doesn't reflect true success is a trap, "when a measure becomes a target, it ceases to be a good measure" (Goodhart's law). The OEC. The Overall Evaluation Criterion is the single, pre-chosen metric (or a principled combination) that best captures long-term value and that you're willing to be judged by. A good OEC is: alig…

One metric to decide, several to keep you honest

Your answer

More Statistics questions