The Horvitz-Thompson Estimator

You're estimating total trading volume across a universe of 3,000 stocks, but you didn't sample them uniformly — large-cap names were deliberately oversampled because they're more liquid and more important to get right, so a stock's chance of landing in your sample varied from name to name. If you just average the sampled stocks' volumes and multiply by 3,000, you'll overstate the total, because your sample is stuffed with large-cap names that aren't representative of the whole universe. The fix isn't to throw out the oversampling — it's to correct for it directly, by giving each sampled observation a weight that undoes exactly how likely it was to be picked in the first place.

An analogy: a raffle with unequal ticket counts

Imagine a raffle where richer participants bought more tickets, so they're more likely to win, and you want to estimate the average wealth of everyone who entered, not just the winners. If you only look at winners' wealth and average it naively, you'll overstate average wealth, because winners were selected precisely because they had more tickets (more chances), which correlates with wealth. To correct this, you'd down-weight each winner in proportion to how many tickets they held — someone with ten times the tickets of another winner, and therefore ten times the chance of winning, should count for only a tenth as much when you're trying to represent the general pool of entrants. That inverse-weighting is exactly the Horvitz-Thompson idea.

The formula, one piece at a time

Suppose each unit $i$ in the population has a known inclusion probability $\pi_i$ — the probability that unit $i$ ends up in your sample, which can differ across units by design (as with deliberate oversampling of large caps). For each unit $i$ that actually lands in the sample, let $y_i$ be its measured value. The Horvitz-Thompson estimator of the population total is

$$\hat Y_{HT} = \sum_{i \in \text{sample}} \frac{y_i}{\pi_i}.$$

In plain English: each sampled observation is scaled up by the inverse of its own chance of being sampled — a unit that had only a 5% chance of selection stands in for about 20 similar unsampled units, so it gets weighted 20 times; a unit that had a 50% chance of selection stands in for only about 2, so it gets weighted twice. This weighting exactly cancels the selection bias baked into an uneven sampling design, and the resulting estimator is unbiased for the true population total regardless of how skewed the inclusion probabilities were, as long as every unit had some nonzero chance of being sampled and you know that chance.

Worked example 1: unequal inclusion probabilities

Three stocks are sampled from a much larger universe. Stock A had inclusion probability $\pi_A = 0.5$ (heavily oversampled, large-cap) and daily volume $y_A = 40$ ($40m). Stock B had $\pi_B = 0.1$ and $y_B = 8$ ($8m). Stock C had $\pi_C = 0.02$ (rarely sampled, small-cap) and $y_C = 1$ ($1m). The Horvitz-Thompson total contribution from these three is

$$\hat Y_{HT} = \frac{40}{0.5} + \frac{8}{0.1} + \frac{1}{0.02} = 80 + 80 + 50 = 210,$$

i.e. $210 million. Notice stock C, despite the smallest raw volume, contributes as much as stock B to the total estimate, because it represents a far larger number of similarly-rare small-cap stocks that weren't sampled at all — its low sampling probability means it has to "stand in" for many unseen peers.

Worked example 2: what happens if you ignore the weights

Using the same three stocks but naively averaging and scaling as if sampling were uniform: average volume $= (40 + 8 + 1)/3 \approx 16.3$ ($16.3m), and if this "represents" a universe of, say, 15 similar stocks, the naive total estimate is $15 \times 16.3 \approx 245$, i.e. $245 million — a different, biased number that has no principled connection to the true population total, because it silently assumes every stock had an equal chance of appearing in the sample when they clearly didn't. The Horvitz-Thompson total of $210m (scaled appropriately to the full universe count) is the one with a guarantee of unbiasedness; the naive average has no such guarantee once sampling probabilities differ.

What this means in practice

The Horvitz-Thompson estimator is the mathematical backbone behind any weighted survey estimate: index reconstitution samples, inverse-propensity-weighted causal inference estimates, and any dataset built by deliberately oversampling a rare or important subgroup and needing to reweight back to a representative total. It's also exactly the logic behind "inverse probability weighting" used to correct for non-random data availability, such as when illiquid stocks are less likely to have clean price data and therefore need up-weighting to avoid understating their share of a universe-level statistic.