Ordering Your Tests Cheapest-Kill-First

Given a new idea, it's natural to start building the most thorough, careful version of the test first — clean data pipeline, proper backtest engine, full universe. That instinct is backwards for most ideas, because most ideas don't survive even a rough test, and the careful version costs far more time to build than a quick one. Ordering tests cheapest-kill-first means running whatever test is fastest to build and would most likely reveal a fatal flaw, before investing in anything more elaborate — the goal at each step is to find the cheapest way to prove the idea wrong, not the most impressive way to prove it right.

Why "trying to kill it" is the efficient order

If an idea has an obvious weak point — a factor that might just be repackaged momentum, a signal that might not survive transaction costs — testing that weak point first, cheaply, either kills the idea in an hour or clears the most likely objection early, before any time is spent on the more expensive parts of the analysis. This is a different mindset from testing in whatever order feels most natural to build; it's specifically testing in the order that gets to "this doesn't work" the fastest, if it's going to happen at all. A team practiced at this spends its expensive, careful testing time almost exclusively on ideas that have already survived several cheap attempts to kill them.

Worked example

A researcher has an idea: buying stocks after a spike in employee reviews on a job-review website predicts positive returns. The expensive version of this test needs a proper backtest engine, careful point-in-time review data, and a full universe. The cheap version, tried first: pull a quick sample of fifty cases where a spike happened, and just look — by eye — at what the stock did over the following month. If those fifty cases show no visible pattern at all, that's often enough to shelve the idea without ever building the full backtest. If they do show something, only then is the more expensive, rigorous version worth building, because now there's a reason to believe the investment will pay off.

What this means in practice

Cheapest-kill-first testing means most of a researcher's ideas die in the first hour rather than the third week, which feels less satisfying in the moment but leaves far more time for the smaller number of ideas that survive the cheap attempts and deserve the expensive, careful treatment. The discipline requires resisting the pull to build the polished version first just because it's more interesting to build — the boring quick check is doing the actual work of protecting the team's time.