Holdout Budget and Peek Accounting

A team is told "don't overfit the test set" but given no way to know whether they have. They've checked it eleven times over three months — was that too many? A holdout set doesn't come with a meter on it, so teams routinely underestimate how much they've already spent. Peek accounting fixes this by treating every look at held-out data as a withdrawal from a finite budget, tracked explicitly, so a team can see exactly how much validity remains before they run out.

Every look costs something

Each time a researcher checks a metric on held-out data — even without changing anything based on it — that look is a decision point, and the decision of whether to keep iterating or stop is itself informed by what was seen. Formal treatments of this (the "reusable holdout" literature) show that the number of times you can safely query a holdout set of a given size, before the reported performance becomes statistically unreliable, is bounded and roughly grows with the square root of the holdout's sample size. In plain English: a bigger holdout buys you more queries, but the relationship is not one-for-one — doubling the holdout size does not double the safe number of peeks, so a small holdout gets exhausted fast.

Peek accounting operationalizes this as a ledger: define a fixed query budget for a holdout set before research starts (say, five checks total), log every actual query against it with a timestamp and requester, and once the budget is spent, the holdout is retired — no more queries, full stop, regardless of how promising the research still looks.

Worked example: a five-query budget

A team allocates a validation set with a budget of five queries for a quarter-long research effort. Query 1 (week 2): checks a baseline model, informs a decision to try feature engineering. Query 2 (week 5): checks after feature engineering, small improvement seen. Query 3 (week 8): checks after a new labeling scheme, meaningful improvement. Query 4 (week 10): checks after hyperparameter tuning, marginal change. Budget remaining: one query. The team must now decide whether to spend the final query confirming the current best configuration, or hold it in reserve for one more promising idea — they cannot do both, and once query 5 is spent, the validation set is retired regardless of whether a sixth idea shows up.

What this means in practice

Peek accounting turns an invisible, easy-to-rationalize habit ("just one more quick check") into a visible, hard constraint that forces discipline about which ideas are worth spending a scarce query on. It also produces an audit trail: if a strategy later underperforms live, the query log shows exactly how the validation set was used and whether the process was followed, which a purely informal "we didn't overfit, trust us" claim cannot provide. Teams that skip peek accounting tend to discover the cost only in production, when a strategy that looked robust in validation decays immediately once real, un-peeked-at data arrives.