Design Effect and Cluster Sampling

You want to estimate the average trade size at a broker by sampling 1,000 trades, but pulling 1,000 fully independent trades is expensive, so instead you sample 50 client accounts and take all 20 trades each of them made that month. You now have 1,000 trade observations — but they are not 1,000 independent pieces of information, because trades from the same account tend to be similar (the same client's trades cluster around their typical size). Treating this cluster sample as if it were 1,000 independent random draws will make your confidence interval look far tighter than it really is. The design effect is the number that tells you exactly how much narrower your interval falsely appears to be, and how to correct it.

An analogy: asking one family member instead of ten strangers

If you want to learn the average height of a town's population, interviewing ten strangers gives you ten mostly-independent data points. Interviewing ten members of the same family instead gives you ten numbers too, but families share genes and diet, so those ten numbers are correlated and collectively carry much less new information than ten strangers would — maybe as little as the information in two or three independent people. The "effective sample size" of that family sample is much smaller than 10, even though you wrote down ten numbers. Cluster sampling has exactly this problem whenever the thing you're measuring is more similar within a cluster than across clusters.

The formula, one piece at a time

Let $m$ be the average number of elements sampled per cluster, and let $\rho$ (rho) be the intraclass correlation — how similar elements within the same cluster are to each other, relative to elements in different clusters, ranging from 0 (no similarity, clusters are as good as random draws) to 1 (every element in a cluster is identical). The design effect is

$$\text{DEFF} = 1 + (m - 1)\rho.$$

In plain English: DEFF tells you the factor by which your cluster sample's variance is inflated compared to a simple random sample of the same total size. If $\rho = 0$, DEFF $= 1$ and clustering costs you nothing. If clusters are highly homogeneous ($\rho$ close to 1) and you take many elements per cluster (large $m$), DEFF grows large, meaning most of your observations are redundant. Your effective sample size — the size of a simple random sample that would give the same precision — is $n_{\text{eff}} = n / \text{DEFF}$, where $n$ is your raw observation count.

Worked example 1: the trade-size survey

With $m = 20$ trades per account and an intraclass correlation of $\rho = 0.3$ (trades within one client's account are moderately similar in size), the design effect is

$$\text{DEFF} = 1 + (20 - 1)(0.3) = 1 + 5.7 = 6.7.$$

Your raw sample is $n = 1{,}000$ trades, so the effective sample size is $n_{\text{eff}} = 1{,}000 / 6.7 \approx 149$. Your 1,000 trade observations carry roughly the statistical information of only 149 independent trades. A confidence interval computed as if you had 1,000 independent draws would be roughly $\sqrt{6.7} \approx 2.6$ times too narrow, badly overstating your precision.

Worked example 2: fewer, deeper vs. more, shallower clusters

Compare two designs with the same 1,000 trades and $\rho = 0.3$ fixed: Design A samples 50 accounts with $m = 20$ trades each (as above, $n_{\text{eff}} \approx 149$). Design B samples 200 accounts with $m = 5$ trades each: $\text{DEFF} = 1 + (5-1)(0.3) = 1 + 1.2 = 2.2$, giving $n_{\text{eff}} = 1{,}000 / 2.2 \approx 455$. Design B, spreading the same 1,000 observations across more, smaller clusters, gives roughly three times the effective sample size of Design A — because fewer redundant observations are drawn from any one correlated cluster. This is why survey designers generally prefer more clusters with fewer elements each over fewer clusters sampled deeply, whenever the sampling cost allows it.

What this means in practice

The design effect matters wherever data is naturally grouped: trades within a client account, transactions within a trading day, loans within a branch, or repeated observations within a stock. Standard errors computed with an ordinary formula (ignoring clustering) are routinely too small by a factor of $\sqrt{\text{DEFF}}$, which can turn a genuinely insignificant result into one that looks statistically significant purely from underestimated variance. Clustered or "sandwich" standard errors, used routinely in panel-data regressions, are the practical fix — they explicitly account for within-cluster correlation rather than pretending it away.