Building an Alternative-Data ML Pipeline

A vendor sells a fund a feed of anonymized credit card transactions: millions of rows a day, each one a merchant, an amount, a timestamp, a rough geography. That feed is not a trading signal. It's raw material, and the distance between raw material and a number a portfolio manager can act on is a whole pipeline of unglamorous work — most of it invisible in a research paper's results section, and most of it where an alt-data strategy actually succeeds or fails.

The stages, in order

Cleaning and deduplication. Raw feeds have gaps, duplicate records, and merchant identifiers that change format mid-stream (the same coffee chain might appear as "SBUX #4471" one week and "STARBUCKS STORE 4471" the next). Getting this wrong doesn't just add noise — it silently double-counts or drops transactions, which shows up later as a phantom signal that vanishes when the vendor fixes their formatting.

Entity resolution. A transaction panel reports merchant-level activity; a trading desk needs company-level revenue. Mapping "SBUX #4471" and "Starbucks Corp online" and forty other merchant strings to the single ticker SBUX — and handling subsidiaries, franchisees, and brands owned by a different public parent — is its own project, and errors here (attributing a franchisee's sales to the wrong parent, or missing a subsidiary entirely) directly corrupt the downstream revenue estimate (see Credit Card Transaction Panels).

Panel scaling. A card panel captures some fraction of total consumer spending — often well under 10%, and that fraction varies by region, merchant type, and time as the card issuer's own customer base shifts. Turning "panel spend at Starbucks rose 8% year over year" into "Starbucks' actual quarterly revenue will be X" requires a scaling model, calibrated against past reported earnings, that itself has an error bar which needs to be respected downstream rather than treated as ground truth.

Feature construction and point-in-time alignment. The final features — year-over-year spend growth, same-store proxies, geographic breakdowns — have to be built using only data that would genuinely have been available on each historical date, not the fully cleaned, fully backfilled version of the dataset as it looks today (see Point-in-Time Data). Vendors routinely restate and backfill their own historical panels, so a naive backtest run on today's "history" is quietly cheating relative to what a trader actually saw in real time.

Worked example

A pipeline ingests a card panel showing Chipotle-tagged transaction volume up 11% year over year for a quarter, from a panel the fund estimates covers about 4% of Chipotle's total card-paying customers. Historical calibration against the last eight reported quarters shows the panel's year-over-year growth number has tracked Chipotle's actual reported same-store sales growth with a correlation of about 0.75 and a typical scaling factor of roughly 0.9 (panel growth tends to run slightly hotter than reported growth). Applying that calibration, the pipeline's point estimate for the upcoming quarter's same-store sales growth is about $11\% \times 0.9 \approx 9.9\%$, with a stated confidence band of roughly ±2 percentage points reflecting the historical tracking error — a specific, testable number a PM can size a position against, not just "the panel looks strong."

What erodes it

The most common way an alt-data pipeline quietly fails is survivorship in the merchant mapping — as a panel's coverage shifts over time (a card issuer gains or loses market share in a region), the "same" year-over-year comparison is actually comparing two different underlying populations, and the resulting drift looks exactly like a real signal until it's traded and doesn't hold up.