Modes of Convergence: In Probability and In Distribution

Statisticians say an estimator "converges" as the sample size grows, but that word hides two genuinely different promises. One says your estimate homes in on a single, specific number. The other says your estimate's shape of uncertainty settles into a recognizable pattern, even though the estimate itself is still random and never stops moving. Confusing the two leads to real mistakes — like thinking a large sample average is a fixed number when it's really a bell curve you've only shrunk, not eliminated.

An analogy: a marksman and a moving target

Convergence in probability is a marksman whose shots cluster tighter and tighter around the bullseye as they get more practice — with enough practice, the chance of landing more than a millimeter from the center goes to zero. The bullseye is a fixed point, and the shots collapse onto it.

Convergence in distribution is different: it's not about hitting a fixed point at all, but about the pattern of where shots land settling into a recognizable, unchanging shape — say, a tight cluster in the shape of a specific oval — even though every individual shot is still random and the cluster never shrinks to a single point. The oval itself stops changing shape; the shots inside it never stop moving.

In probability, one symbol at a time

An estimator $\hat\theta_n$ (read "theta-hat, based on $n$ observations") converges in probability to the true value $\theta$ if, for every tiny tolerance $\epsilon$ you name,

$$P\big(|\hat\theta_n - \theta| > \epsilon\big) \to 0 \quad \text{as } n \to \infty$$

In plain English: name any margin of error, however small, and the chance your estimate misses the truth by more than that margin shrinks to zero as your sample grows. This is written $\hat\theta_n \xrightarrow{p} \theta$, and it is exactly the guarantee behind the Law of Large Numbers: a sample mean converges in probability to the true population mean.

In distribution, one symbol at a time

A sequence of random variables $Z_n$ converges in distribution to a random variable $Z$ if the probability that $Z_n$ falls below any value $z$ gets arbitrarily close to the probability that $Z$ falls below that same $z$:

$$P(Z_n \le z) \to P(Z \le z) \quad \text{for every } z$$

In plain English: the whole shape of $Z_n$'s distribution — not $Z_n$ itself — comes to resemble the shape of $Z$'s distribution. $Z_n$ is still random and still bouncing around; only its histogram settles down. This is written $Z_n \xrightarrow{d} Z$, and it is the guarantee behind the Central Limit Theorem: the standardized, rescaled sample mean converges in distribution to a standard normal — even though the raw sample mean itself is converging in probability to a single point, not a spread-out normal curve.

The explorer above shows both ideas at once. Watch the sample mean tighten toward the true parent mean as $n$ grows — that's convergence in probability, collapsing onto a point. Now imagine instead plotting $\sqrt{n}$ times the mean's deviation from the truth — that quantity does not collapse; its histogram settles into a fixed bell shape. Same underlying process, two different questions about it.

The running-average explorer above is convergence in probability in its purest form: the line homes in on one fixed number and the band around it visibly narrows as $n$ grows — there is no analogous picture for convergence in distribution, because what "narrows" there is not the spread of $Z_n$ but the gap between its histogram and a fixed target shape.

Worked example 1: watching a sample mean converge in probability

Roll a fair six-sided die repeatedly; the true mean is $\mu = 3.5$. Simulate averages over growing sample sizes: after $n=10$ rolls, a plausible sample mean is $3.9$, missing the truth by $0.4$. After $n=100$ rolls, a plausible mean is $3.58$, missing by $0.08$. After $n=10{,}000$ rolls, a plausible mean is $3.503$, missing by $0.003$.

Fix a tolerance, say $\epsilon = 0.1$. At $n=10$, missing by more than $0.1$ is quite likely — the sample mean regularly lands more than 0.1 away from 3.5. By $n=10{,}000$, the chance of missing by more than $0.1$ has become vanishingly small, even though individual outcomes are still random. That shrinking probability of a "large miss," for any tolerance you name, is convergence in probability, made concrete.

Worked example 2: the CLT's rescaled quantity that never collapses

Take the same die, but now track $Z_n = \sqrt{n}\,(\bar{X}_n - 3.5)$ — the deviation from the truth, blown back up by $\sqrt{n}$ to compensate for the shrinking. The true variance of one die roll is $\sigma^2 = 2.917$, so the CLT predicts $Z_n \xrightarrow{d} N(0, 2.917)$.

At $n=100$: $\bar X_{100} = 3.58$ gives $Z_{100} = \sqrt{100}\times 0.08 = 0.8$. At $n=10{,}000$: $\bar X_{10000}=3.503$ gives $Z_{10000} = \sqrt{10000}\times 0.003 = 0.3$. These two values of $Z_n$ look similarly small and unremarkable — because unlike $\bar X_n$ itself, $Z_n$'s spread is not shrinking with $n$; both are draws from roughly the same bell curve of width $\sqrt{2.917}\approx 1.71$. Run this simulation many times at each $n$ and histogram the resulting $Z_n$ values: at $n=100$ the histogram is already close to a $N(0, 2.917)$ bell curve, and at $n=10{,}000$ it looks essentially identical — the shape stopped changing, even though no individual $Z_n$ value ever stops being random.

What this means in practice

• Consistency of an estimator — the property that it converges to the truth as data grows — is exactly convergence in probability, and it's the minimum bar any usable estimator must clear.
• Confidence intervals and hypothesis tests rely on convergence in distribution, not convergence in probability — the whole reason a $t$-statistic has a known, usable distribution in large samples is that it converges in distribution to a normal, even though the statistic itself never stops being random.
• "Large sample" claims in a paper or model should specify which kind of convergence is being invoked — a claim that a parameter estimate is "accurate in large samples" needs convergence in probability; a claim that "the test statistic is approximately normal in large samples" needs convergence in distribution. They are not interchangeable justifications.