Golden Master and Regression Tests

Some systems are too complicated to write a normal unit test for. A full backtest pipeline takes years of market data, runs thousands of decision points, and produces a report with hundreds of numbers — nobody is going to hand-derive the "correct" value of every one of those numbers to assert against. But you still need to know if a refactor silently changed behavior. Golden master testing sidesteps the problem: run the system once, save its full output as the "golden" reference, and afterward simply check that a new run's output still matches the golden file.

The idea: snapshot the output, diff against it forever after

The technique, sometimes called a "characterization test" or "snapshot test," has three steps:

1. Capture. Run the current system on a fixed, realistic input and save the full output — a CSV of daily P&L, a JSON blob of computed risk metrics, a rendered report — to a file checked into version control.
2. Compare. On every future run (in CI, on every commit), rerun the same input through the current code and diff the fresh output against the saved golden file.
3. Update deliberately. When a diff appears, a human decides: is this an unintended regression (fix the code), or an intentional change (regenerate the golden file and commit it, ideally with a code review explaining why the numbers moved)?

The critical property this gives you is not correctness — the golden file was never proven "correct" in the mathematical sense, it's just whatever the system produced on day one. What it gives you is stability under refactoring: if you rewrite the internals of a pricing engine for speed and the golden master still matches, you have strong evidence the rewrite preserved behavior, without needing to re-derive every expected number from first principles.

Worked example 1: capturing a golden master for a backtest summary

import json

def run_backtest_summary(strategy, prices):
    return {
        "total_return": strategy.total_return(prices),
        "sharpe": strategy.sharpe(prices),
        "max_drawdown": strategy.max_drawdown(prices),
        "num_trades": strategy.num_trades(prices),
    }

# one-time capture, run manually and reviewed before committing
summary = run_backtest_summary(MyStrategy(), fixed_test_prices)
with open("golden/backtest_summary.json", "w") as f:
    json.dump(summary, f, indent=2)

Suppose this produces {"total_return": 0.184, "sharpe": 1.12, "max_drawdown": -0.093, "num_trades": 47}. That file is committed to the repo as the reference — not because those numbers were independently verified against a textbook, but because they represent the strategy's behavior on a known, fixed dataset at a known point in time.

Worked example 2: the regression check, and a real diff

import numpy as np

def test_backtest_matches_golden_master():
    with open("golden/backtest_summary.json") as f:
        golden = json.load(f)

    fresh = run_backtest_summary(MyStrategy(), fixed_test_prices)

    for key in golden:
        np.testing.assert_allclose(fresh[key], golden[key], rtol=1e-6,
                                    err_msg=f"{key} drifted from golden master")

Say a teammate refactors how the strategy handles a stock split and, unnoticed, the fix changes num_trades from 47 to 52 and sharpe from 1.12 to 0.97. The golden master test fails immediately, naming exactly which fields moved and by how much — pointing straight at the corner cases, split handling in this case, that the change touched. Without the golden master, this shift could have shipped silently; the individual unit tests for order placement or Sharpe computation might all still pass in isolation, because the bug is in how they interact over a full run, not in any one function.

What this means in practice

Golden master tests are cheap to write (capture once) but require discipline to maintain: every intentional behavior change means regenerating and re-reviewing the golden file, which can become a rubber-stamp habit ("just update the golden file, tests will pass") if the team isn't careful. Keep the fixed input small and realistic enough that a reviewer can actually eyeball a diff and judge whether a changed number makes sense.

Where it shows up

In interviews, this comes up as "how would you make sure a big refactor didn't change behavior" for a system too large to unit-test exhaustively — golden master testing is the expected answer. In production, quant teams run golden master checks on backtest engines, report generators, and full pricing libraries specifically because a subtle regression in aggregate output (a shifted Sharpe ratio, a changed set of trades) is exactly the kind of bug that individual unit tests, each passing on its own, can completely miss.