Point-in-Time Corpora for LLM Backtests

Suppose you want to backtest a strategy that has an LLM read a company's 10-K and generate a trading signal, tested over the last ten years of filings. If your document store simply holds the current, latest version of every filing indexed by company, you have a subtle but fatal problem: some filings get restated after the original filing date — a revenue figure corrected, a footnote amended — and if your backtest retrieves the restated version to simulate a trading decision made before the restatement happened, the LLM is reading information nobody could have had at the time. The strategy looks great in backtest and can fail badly live, because the backtest was silently reading the future.

What "point-in-time" adds to a document corpus

A point-in-time corpus stores not just the content of each document but the full history of what was known and when — every filing version tagged with the date it was actually published, and ideally a record of any later amendments as separate, dated versions rather than overwrites. A query for "the 10-K available for company X as of March 15, 2019" must return exactly the version that existed then, not a version corrected the following year. This is a stricter requirement than most NLP datasets are built to satisfy, because standard scraped corpora typically keep only the latest snapshot of each document.

Two specific hazards this guards against

Restatement leakage: a filing amended after the fact (common after an accounting error) can silently replace the original in a naively-updated database. A backtest reading the amended version at an earlier simulated date sees a cleaner, corrected picture the market never had — inflating backtested performance around exactly the events that matter most, since restatements often follow bad news.

Release-timing leakage: filings and news often have a lag between the event date and the public release date — an 8-K filed the morning after a board decision, or an earnings call transcript posted hours after the call ends. If the corpus is indexed by event date rather than public availability, a backtest can retrieve a document "as of" a date before it actually existed, again handing the strategy information it never could have had live.

Worked example: a restated earnings figure

A company files its 10-K on February 10, reporting quarterly earnings per share of $1.20. On August 3 of the same year, it files a restatement correcting an accounting error, revising that EPS figure down to $0.85. A backtest simulating a trading decision on March 1 — after the original filing but before the restatement — must retrieve the $1.20 figure, because that's what was public knowledge on March 1; a corpus that only keeps the latest version would hand the backtest $0.85 instead, changing the LLM's read of the company's health for a date at which the correction did not yet exist. Only a point-in-time corpus, storing both versions with their respective publication dates, gets this right.

What this means in practice

Building a point-in-time corpus is significantly more engineering work than scraping the latest version of every document — it means storing full revision histories with accurate publication timestamps, not just content. Skipping it doesn't cause an obvious crash; it causes a backtest that quietly overstates performance in exactly the situations (restatements, corrections, delayed disclosures) that a live strategy would have had to face without the benefit of hindsight.