The Law of Large Numbers

The law of large numbers is the theorem that makes empirical work possible: it says the average of many independent draws converges to the true mean. Every casino edge, every insurance book, every Monte Carlo estimate, and every claim that a strategy with positive expectation "wins in the long run" rests on it. But the honest version comes with fine print, which mode of convergence, how fast, and the conditions under which it silently fails, and that fine print is exactly where undisciplined intuition loses money.

Two theorems, two modes of convergence

Let $X_1, X_2, \dots$ be i.i.d. with mean $\mu = \mathbb{E}[X_i]$, and write the sample mean $\bar X_n = \frac{1}{n}\sum_{i=1}^n X_i$.

Weak Law (WLLN). $\bar X_n$ converges to $\mu$ in probability: for every $\varepsilon > 0$, $\lim_{n\to\infty}\mathbb{P}\big(|\bar X_n - \mu| > \varepsilon\big) = 0.$ For any fixed tolerance, the probability that the current average is off by more than $\varepsilon$ vanishes as the sample grows.

Strong Law (SLLN). $\bar X_n$ converges to $\mu$ almost surely: $\mathbb{P}\Big(\lim_{n\to\infty}\bar X_n = \mu\Big) = 1.$ Almost every individual sample path of running averages settles down to $\mu$ and stays there. This is genuinely stronger: the WLLN allows the average to wander outside $[\mu-\varepsilon,\mu+\varepsilon]$ infinitely often (each such excursion just becoming rarer); the SLLN forbids it beyond some finite point on almost every path. The SLLN holds under only $\mathbb{E}|X_i| < \infty$ (Kolmogorov); the WLLN can be pushed even weaker.

Proof sketch via Chebyshev (the finite-variance WLLN)

Assume additionally $\operatorname{Var}(X_i) = \sigma^2 < \infty$. Then by linearity and independence Expectation, Variance & Moments, $\mathbb{E}[\bar X_n] = \mu, \qquad \operatorname{Var}(\bar X_n) = \frac{\sigma^2}{n}.$ The variance of the average shrinks like $1/n$, this single fact is the law of large numbers. Chebyshev's inequality then bounds the deviation directly: $\mathbb{P}\big(|\bar X_n - \mu| > \varepsilon\big) \le \frac{\operatorname{Var}(\bar X_n)}{\varepsilon^2} = \frac{\sigma^2}{n\varepsilon^2} \xrightarrow{n\to\infty} 0.$ That is the whole WLLN under finite variance, in two lines. The SLLN needs more (a fourth-moment Borel–Cantelli argument, or Kolmogorov's truncation for the general case), but the mechanism is the same collapsing variance. The Moment Generating Functions machinery sharpens this: large-deviation bounds show the tail probability actually decays exponentially in $n$, not just like $1/n$.

The rate is the catch: $1/\sqrt{n}$

The LLN says the average converges; it does not say convergence is fast. Since $\operatorname{Var}(\bar X_n) = \sigma^2/n$, the standard deviation of the estimate, the typical error, is $\text{SE}(\bar X_n) = \frac{\sigma}{\sqrt{n}}.$ Error falls like $1/\sqrt n$, so cutting the error in half requires quadrupling the sample. This square-root wall is the governing economics of every simulation and every backtest. It is also the bridge to the The Central Limit Theorem, which describes the distribution of the leftover $\sqrt n(\bar X_n - \mu)$ error rather than merely asserting it shrinks.

Worked example, Monte Carlo pricing accuracy

You price a derivative by simulating $n$ payoffs and averaging; the estimator is $\bar X_n$ and its error is $\sigma/\sqrt n$. Suppose the payoff has standard deviation $\sigma = 20$ and you want the pricing standard error below $0.01$. Then you need $\frac{20}{\sqrt n} \le 0.01 \;\Longrightarrow\; \sqrt n \ge 2000 \;\Longrightarrow\; n \ge 4\times 10^6$ simulations. Want one more digit (error $0.001$)? That is $4\times 10^8$, a hundredfold more work. This is why variance reduction (antithetic variates, control variates, importance sampling) is not a nicety but the core craft of Monte Carlo: since you cannot beat $1/\sqrt n$, you attack the $\sigma$ in the numerator instead.

Failure modes and subtleties

• Infinite mean or variance kills it. For a Cauchy (Student-$t$ with 1 d.o.f.), $\mathbb{E}|X| = \infty$: the sample mean $\bar X_n$ has the same Cauchy distribution for every $n$ and never converges. Averaging heavy-tailed data can be worthless, a sobering fact when return tails may have undefined higher moments. Always check the moment exists before invoking the LLN.
• It says nothing about finite samples. "The LLN guarantees I'll profit eventually" ignores that "eventually" can be past your capital's survival. A positive-edge bettor still goes broke if bet sizing ignores the path, this is exactly why the The Kelly Criterion optimises geometric growth rather than trusting the LLN to bail out reckless sizing.
• The gambler's fallacy is the LLN misread. The average converging does not mean deviations get "corrected"; past outcomes do not pull future ones. Convergence happens because new draws dilute old deviations, not because a debt is repaid. After ten heads, the eleventh is still 50/50.
• Independence and identical distribution. Serial dependence (autocorrelated returns, regime shifts) breaks the i.i.d. premise; the average may converge to something other than the current mean, or converge far slower. Ergodic theorems extend the LLN to stationary dependent sequences, but non-stationary financial data can violate even those.
• Bias survives the LLN. The LLN removes variance, not bias. A systematically mismeasured payoff converges precisely to the wrong number, with growing false confidence.

In interviews

The LLN itself is rarely the whole question; the discrimination is whether you know the WLLN/SLLN distinction (in probability vs. almost surely) and, above all, the $1/\sqrt n$ rate. Expect the Monte Carlo sizing question: "how many simulations to halve the error?", quadruple. Expect the gambler's-fallacy trap dressed up ("a stock is down ten days running, is it 'due'?"), no, and explain that convergence is by dilution. A sharp interviewer will probe the failure case: "when does averaging not help?", heavy tails with undefined mean (Cauchy) is the crisp answer. Tie it forward to the The Central Limit Theorem, which upgrades "the error shrinks" to "the error is asymptotically normal with known scale", the result that actually lets you put a confidence interval on $\bar X_n$.