Signals From Sparse, Intermittent Data

Patent filings happen a handful of times a year per company, if that. Executive departures happen rarely, and for most companies, never in a given quarter. Lawsuits, product recalls, activist-investor letters — a lot of genuinely informative events are, by their nature, sparse: absent for the overwhelming majority of company-days and present in a short, irregular burst when they do occur. That structure breaks most of the usual signal-construction toolbox, which was built assuming something closer to a number every day for every name.

Why the usual tools fail here

Z-scoring assumes a reasonably full cross-section every period; with sparse data most of the cross-section is missing on any given day, so a z-score computed only over the handful of names with an observation is comparing a tiny, non-random subset against itself. Averaging over a rolling window helps smooth out gaps, but the right window length depends entirely on how frequently the underlying event occurs — a rolling 5-day window is nearly always empty for an event that happens twice a year.

Worked example

A researcher studies patent-filing counts as a predictor of future R&D-driven returns for a universe of 900 small-cap technology companies. In a typical month, only about 60 of the 900 names file any patent at all. Rather than z-scoring the monthly count across all 900 names (which would compare 60 real observations against 840 zeros, mechanically making any filer look extreme), the researcher instead restricts the cross-section each month to just the 60 filers and ranks filing intensity only among them, separately tracking "did this company file at all this month" as its own binary feature. The two-feature approach — occurrence, then intensity conditional on occurrence — outperforms the single z-scored count by a wide margin in an out-of-sample test.

What this means in practice

Model sparse data as two questions, not one: whether an event happened at all (often the more informative half), and, conditional on it happening, how large or notable it was. Trying to collapse both into a single continuous score usually loses information the split representation would have kept.