The Sampling Distribution

Here is the idea almost everything in statistics is built on, and the one beginners most often skip. When you collect a sample and compute a number from it, say the average, that number is itself random. Draw a different sample and you get a slightly different average. The sampling distribution is the distribution of that number across all the samples you could have taken. You only ever hold one sample in your hands, so you never see this distribution directly, but its shape is what tells you how much to trust your estimate.

Keep two distributions clearly apart. One is the spread of the raw data, how much individual values scatter (that's the ordinary standard deviation $\sigma$). The other is the spread of a summary like the mean, computed over many hypothetical samples. The second is far narrower, because averaging cancels out the ups and downs. That narrow curve is the sampling distribution of the mean, and its width has a name: the standard error.

Three facts pin the sampling distribution of the mean down:

• Center. It sits right on the true value $\mu$. The sample mean is unbiased: on average it neither overshoots nor undershoots. See Bias, Variance, and the Quality of Estimators.
• Spread. Its standard deviation, the standard error, is $\operatorname{SE} = \sigma / \sqrt{n}$. More data ($n$) means a tighter curve, but only as the square root of $n$.
• Shape. For a large sample it's approximately a bell curve no matter what the population looks like, that's the The Central Limit Theorem doing the work.

Worked example: one value vs the average of a hundred

Suppose a population has standard deviation $\sigma = 20$ (think test scores, or a noisy daily return in some unit). A single observation typically lands within about $\pm 2\sigma = \pm 40$ of the mean, that's a wide, useless-on-its-own range.

Now take a sample of $n = 100$ and average it. The standard error is

$$\operatorname{SE} = \frac{\sigma}{\sqrt{n}} = \frac{20}{\sqrt{100}} = 2 .$$

So the sample mean lands within about $\pm 2 \times \operatorname{SE} = \pm 4$ of $\mu$ roughly 95% of the time, ten times tighter than a single value, even though the raw data are just as noisy as ever. Want to halve that error to $\pm 2$? You need $\operatorname{SE} = 1$, which means $\sqrt{n} = 20$, i.e. $n = 400$. Four times the data to double the precision is the recurring tax of the $\sqrt{n}$ law, and it's why estimating a mean return to real precision takes absurdly long samples.

Why it's the engine of inference

Every confidence interval and every hypothesis test is secretly a statement about the sampling distribution. A 95% confidence interval is just "reach out about $\pm 2$ standard errors from the estimate," because that's where the sampling distribution puts 95% of its mass. A test asks "is my observed statistic way out in the tail of the sampling distribution it would have under the null?" Get the sampling distribution right and the rest of inference is bookkeeping.

Common pitfalls

• Confusing the two spreads. $\sigma$ (standard deviation) describes how data scatter; $\operatorname{SE} = \sigma/\sqrt{n}$ describes how the estimate scatters. Reporting $\sigma$ when you mean $\operatorname{SE}$ (or vice versa) is a classic slip, and off by a factor of $\sqrt{n}$.
• The data histogram is not the sampling distribution. Plotting your one sample shows the population's shape (spread $\sigma$), not the much narrower distribution of the mean. Don't read precision off the raw histogram.
• Diminishing returns. Because the width falls only as $\sqrt{n}$, doubling accuracy costs four times the data. Chasing a tiny standard error can be wildly expensive. See The Law of Large Numbers.
• Non-independent data break the formula. $\sigma/\sqrt{n}$ assumes independent draws. Autocorrelated returns (momentum, overlapping windows) make the true standard error larger, so naive error bars are too optimistic.

The mental move to install: after any estimate, don't stop at the number, picture the whole curve of numbers you might have gotten, centered on the truth with width $\sigma/\sqrt{n}$. That picture is what turns a point estimate into an honest statement about uncertainty.