Clicks up, page speed down: do you ship?

Question

You test a new search-ranking model. The primary metric, click-through rate, is up a statistically significant +3%. But a guardrail metric, median page-load time, regressed by a significant +120 milliseconds, and another guardrail, the rate of searches with zero clicks, ticked up slightly but not significantly.

Should you ship, and how should you reason about the guardrail regression?

Accepted Answer

What guardrails are for. The primary metric measures the win you are chasing; guardrail metrics protect things you are unwilling to harm even in pursuit of that win, like speed, error rates, revenue, or long-term retention. The whole point of a guardrail is that a primary win does not automatically justify shipping if a guardrail breaks. Do not just celebrate the +3%. A higher click-through rate paired with slower pages is a real trade-off, not a free lunch. Slower pages are known to reduce engagement and revenue over time, and those harms may not show up inside a short test window. There is …

Clicks up, page speed down: do you ship?

Your answer

More Statistics questions