Panel Drift and Vendor Methodology Changes

An alternative-data vendor's "panel" — the set of consumers, credit cards, devices, or locations feeding a dataset — is never a fixed, stable sample. It's a living, growing thing that the vendor is constantly renegotiating, expanding, and adjusting behind the scenes, and none of that internal churn is visible from the outside unless you go looking for it. Panel drift is the umbrella term for these unannounced changes, and it's one of the most common and least discussed ways an alt-data signal that looked great historically stops working (or starts working for the wrong reason) once it's live.

The most basic form of drift is simple panel growth: a credit card panel that covered 2 million cardholders in its first year of operation might cover 8 million three years later, as the vendor adds new bank partnerships. A model trained across that history is implicitly training on a panel whose composition and representativeness kept changing under it — early years might skew toward one region or income bracket the vendor's initial bank partner served, later years correct that skew as new partners are added, and a signal that looks like a genuine trend in consumer behavior can actually just be the panel's demographic mix shifting toward the true population average over time.

A second, sharper form is a discrete methodology change: a vendor switches its underlying data source, changes how it de-duplicates records, updates its algorithm for mapping raw transactions to specific merchants, or loses (or gains) a major data-supplying partner overnight. These show up as a sudden level shift in the data — a company's "sales estimate" derived from the panel might jump 15% quarter over quarter for no economic reason at all, purely because the vendor's coverage of that merchant changed. A model that isn't watching for this will read the jump as a real business signal and trade on it.

For example, a satellite-imagery vendor upgrades its image-processing algorithm for counting cars in retail parking lots. The new algorithm is more accurate, but every historical count in the vendor's database from before the upgrade is on a slightly different scale than every count after it. A trading signal built on quarter-over-quarter changes in parking lot traffic would see an artificial jump exactly at the upgrade date, indistinguishable at first glance from a genuine surge in store visits.

What this means in practice

Guarding against panel drift means monitoring the input data itself, not just the model's output: tracking the raw panel size, its geographic and demographic composition, and its coverage of the specific companies or categories you trade over time, and treating any vendor communication about "backend improvements" as a signal to re-check whether history before and after the change is still comparable. Any backtest built on a long history of alternative data should be treated with extra skepticism if the vendor can't clearly document how their panel and methodology looked at each point in that history, since a backtest run on a panel that grew and changed shape behind the scenes is not really the same experiment repeated over and over.