Aggregating Raw Records Into a Company Panel
Raw data usually arrives as individual events — receipts, shipments, reviews — and someone has to decide how to roll thousands of those events up into one number per company per day.
Prerequisites: Entity Resolution When Identifiers Do Not Match
A satellite-imagery vendor does not sell you "Walmart's foot traffic." It sells you a raw table with one row per parking lot, per image, per day — thousands of individual observations that happen to belong to stores that happen to belong to one company. Turning that raw event table into a single, usable time series — one row per company per date — is the aggregation step, and the choices made here (which stores count, how missing images are handled, how outlier images are treated) shape the resulting signal at least as much as anything that happens afterward.
Aggregation is not a mechanical rollup, it is a modeling decision. Summing, averaging and weighting the same raw records by store size will each produce a genuinely different company-level series, and none of them is the objectively "correct" one.
Choosing how to roll up
For a company on date , built from raw location-level records :
In words: the company-level value is a weighted sum of its individual locations' values on that date, where the weight decides how much each location counts — equal weight per location, weight by store square footage, or weight by that location's typical historical traffic. Changing the weighting scheme is a real research decision, not a cosmetic one, since a handful of very large flagship stores can otherwise dominate the whole company signal.
Worked example
A retailer has 4,200 stores; a satellite feed captures usable images for 3,100 of them on a given day, with coverage skewed toward larger urban locations. An unweighted average of the 3,100 available store-visit counts implicitly overweights the urban subset relative to the company's true store mix, which is 60% suburban and exurban. Reweighting by each store's share of the company's total historical square footage — rather than simply averaging whichever stores happened to have a clear satellite image that day — cuts the correlation between the resulting signal and "fraction of images that happened to be urban" from 0.41 to 0.06, a sign the reweighted series is tracking the company rather than a coverage artifact.
What this means in practice
Document the aggregation formula as carefully as any model specification, and test sensitivity to it — rerun the research with an alternative, reasonable weighting scheme and check whether the signal's performance survives. If a result only holds under one specific rollup choice out of several equally defensible ones, treat that as a warning sign rather than a footnote.
Missing raw records are rarely missing at random — cloud cover blocks images of stores in rainy regions more often, and unweighted averages over "whatever data showed up today" silently reweight the company toward whichever locations happened to have clear skies.
Related concepts
Practice in interviews
Further reading
- Kolanovic and Krishnamachari, 'Big Data and AI Strategies' (J.P. Morgan)