Evaluating a New Dataset: The Full Checklist

A vendor pitches a dataset with a slide showing a back-tested Sharpe ratio of 2.1. That number is the last thing worth checking, not the first. Most datasets that fail, fail on structural grounds that have nothing to do with predictive power — grounds that, once discovered, make the vendor's Sharpe ratio irrelevant regardless of how it was computed. A researcher who checks those grounds first saves weeks spent building a signal on data that was never usable to begin with.

The checklist, in the order that kills the most candidates fastest

Timestamp integrity. Is the field stamped with the date it was actually knowable, or the date it describes? A dataset stamped to a fiscal quarter-end, when the actual data wasn't available until six weeks later, will produce a back-test riddled with look-ahead the moment you use it naively. This single check kills more candidate datasets than any other, and it should be verified against the vendor's own delivery logs, not assumed from documentation.

Coverage. What fraction of your actual tradeable universe does this dataset touch, and how does that fraction change over time? A dataset covering 90% of large caps but 20% of the small-cap names your strategy actually trades is not a 90%-useful dataset for you — it is closer to unusable for anything outside large caps, and the 20% that is covered may not be a random, unbiased 20%.

History and survivorship. How far back does the data go, and does it include companies that later delisted, went bankrupt, or were acquired? A dataset that only contains currently-listed companies is survivorship-biased by construction — any back-test built on it will look better than a live version of the same strategy ever could, because the losers have been silently removed from the sample.

Revision behaviour. Does the underlying data get restated after initial release (common for financial and economic data)? If so, a naive back-test using final, revised values instead of the values available in real time is another form of look-ahead, distinct from the timestamp issue but just as damaging.

Redundancy against what you already own. Does this dataset add information beyond what's already in the research library, or is it a repackaged version of a factor already covered? This is a cost-benefit question, but it belongs on the checklist before serious research time is spent, not after.

A worked scorecard

A vendor offers a supply-chain relationship dataset — which companies are customers/suppliers of which others, updated quarterly.

Check	Finding	Verdict
Timestamp integrity	Relationships stamped to quarter-end, but sourced from filings that lag by up to 10 weeks	Needs an as-available lag correction before use
Coverage	78% of Russell 1000, 12% of Russell 2000	Usable for large-cap strategies only
History	Back to 2015 only	Limits back-test length; no pre-2015 crisis period to test against
Survivorship	Includes delisted/acquired companies as of each historical date	Passes
Revision behaviour	Relationships occasionally revised retroactively when filings are corrected	Minor risk; use point-in-time snapshots, not the latest table
Redundancy	Distinct from existing factor library — no close substitute	Adds new information

This dataset is not disqualified, but the verdict is far more specific than "good" or "bad": usable for a large-cap strategy, with a mandatory timestamp correction, over a limited nine-year history, using point-in-time snapshots rather than the live table. That is the level of specificity a checklist produces and a Sharpe-ratio slide never will.