Reading a Vendor's Own Backtest

A vendor selling a dataset or signal will almost always show you a backtest, and it's rational for it to be their single best-looking result — which is exactly why it needs more scrutiny than a backtest you ran yourself. The vendor has typically tried many variants of the signal, many universes, and many time periods before settling on the one they present, so the number in front of you has already survived an unseen selection process that inflates its apparent quality.

Practical checks: ask what the sample period is and whether it includes a genuinely out-of-sample stretch after the signal was finalized, not just a split of the same historical data used to design it; ask how many other variants were tried before this one was chosen, since that's the real multiple-testing burden even if none of it appears in the report; and ask for the raw signal or data itself so you can run your own backtest with your own costs, universe, and timing assumptions rather than trusting the vendor's.

A vendor might present a backtest showing a Sharpe ratio of 1.8 over five years, but on inquiry reveal that the underlying signal was one of over a hundred variants tested and that the backtest excludes bid-ask spread and market impact costs — reasonable follow-up questions that can turn an eye-catching 1.8 into something closer to 0.6-0.8 once trading costs and a fair accounting for the multiple-testing behind the selection are applied.