Modes of Convergence for Random Variables

Every result you rely on has a limit hiding in it. The law of large numbers says a sample mean settles on the truth. The central limit theorem says a standardised sum settles on a bell curve. Your Monte Carlo pricer says more paths give a better answer. All three say "settles", and none of them mean the same thing by it — so feeding one theorem's notion of settling into another's is a mistake that is hard to see and easy to ship.

Four ways a class can improve

Picture a large class taking a spelling test every week, forever. In week $n$, student $\omega$ scores an error count $X_n(\omega)$. Ask in four ways whether the class is "getting to zero errors".

1. Every student personally gets there and stays. Follow one student down the years: at some point they stop making mistakes and never make another. This is almost sure convergence — about individual life stories.
2. At any given moment, hardly anyone is failing. In week 1000 the fraction with errors is tiny, but no individual is promised anything: a given student might relapse, briefly, forever. This is convergence in probability — about snapshots.
3. The average squared error shrinks. Square everyone's error count and average over the class; that number goes to zero. This is convergence in mean square ($L^2$), unforgiving of rare disasters: one student scoring a million ruins the average even if everyone else is perfect.
4. The shape of the results settles. The weekly histogram stops changing, even though which student holds which score keeps shuffling. This is convergence in distribution — about the pattern, not anybody in it.

Everything below is those four sentences in symbols.

The definitions

Throughout, $X_1, X_2, X_3, \dots$ is a sequence of random variables, $X$ is the target, $\omega$ is one scenario (one student), $n$ is the step number, and $\varepsilon$ (epsilon) is any tolerance you name, however small.

Almost surely ($X_n \to X$ a.s.):

$$P\!\left( \lim_{n \to \infty} X_n = X \right) = 1 .$$

The set of scenarios in which the sequence genuinely converges has probability one. The limit is taken inside, scenario by scenario.

In probability ($X_n \xrightarrow{P} X$): for every $\varepsilon > 0$,

$$\lim_{n \to \infty} P\big( |X_n - X| > \varepsilon \big) = 0 .$$

However tight a tolerance you name, the chance of being outside it at step $n$ shrinks to zero. The limit is now outside, on the probability.

In mean square ($L^2$):

$$\lim_{n \to \infty} E\!\left[ \, |X_n - X|^2 \, \right] = 0 .$$

The average squared miss shrinks to zero. This is the one that mean-squared-error and variance arguments use.

In distribution ($X_n \xrightarrow{d} X$): writing $F_n$ for the cumulative distribution function of $X_n$ and $F$ for that of $X$,

$$F_n(x) \longrightarrow F(x) \quad \text{at every } x \text{ where } F \text{ is continuous.}$$

The probability of landing below any level settles down. Note the variables need not live on the same probability space at all — only their laws are compared.

How they rank

Convergence in distribution is the weakest of the four: implied by all the others, implying none. It is what the central limit theorem delivers. The strong law of large numbers delivers almost sure convergence and the weak law delivers only convergence in probability — that is the entire difference between the two versions.

Worked example 1: in probability but never almost surely

Let $\omega$ be drawn uniformly from $[0,1]$ — think of it as picking one student at random. Sweep a block across the interval in rounds, setting $X_n = 1$ inside the current block and 0 outside:

• Round 1: $X_1 = 1$ on all of $[0,1]$.
• Round 2: $X_2 = 1$ on $[0, \tfrac{1}{2}]$; $X_3 = 1$ on $[\tfrac{1}{2}, 1]$.
• Round 3: $X_4$ on $[0,\tfrac{1}{3}]$; $X_5$ on $[\tfrac{1}{3},\tfrac{2}{3}]$; $X_6$ on $[\tfrac{2}{3},1]$.
• Round $k$: $k$ blocks of width $\tfrac{1}{k}$, together covering the whole interval.

Does it converge in probability to 0? Take any tolerance, say $\varepsilon = 0.5$. Then $P(|X_n - 0| > 0.5) = P(X_n = 1)$, the width of the current block. Rounds finish at steps 1, 3, 6, 10, 15, 21, 28, 36, 45, 55, 66, 78, 91, 105, so $n = 100$ sits in round 14 with width $1/14 \approx 0.0714$, and $n = 1000$ sits in round 45 with width $1/45 \approx 0.0222$. These go to zero, so yes: $X_n \xrightarrow{P} 0$.

Does it converge almost surely to 0? Fix a scenario, say $\omega = 0.6$. Round 1's block covers everything, so $X_1(0.6) = 1$. Round 2's block $[\tfrac{1}{2},1]$ contains $0.6$, so $X_3(0.6) = 1$. Round 3's block $[\tfrac{1}{3},\tfrac{2}{3}]$ contains $0.6$, so $X_5(0.6) = 1$. Every round covers the whole interval, so every $\omega$ is hit exactly once per round, forever: the sequence $X_n(0.6)$ reads $1, 0, 1, 0, 0, 1, 0, \dots$ and never settles. The probability of convergence is $0$, not $1$. No almost sure convergence.

In one line: in probability means the failures get rare; almost surely means the failures stop. Rare is not finished.

The friendly direction: a running average of independent draws does settle permanently, for essentially every path. Add samples in the explorer below and watch the running mean stop wandering, while the shaded $1/\sqrt{n}$ envelope shows the rate at which the "in probability" statement tightens.

Worked example 2: almost surely but not in mean square

Now the other gap. Define $X_n$ to be the number $n$ with probability $1/n^2$, and $0$ otherwise. Work out the two quantities directly:

$n$	$P(X_n = n)$	$E[X_n] = n \cdot \frac{1}{n^2}$	$E[X_n^2] = n^2 \cdot \frac{1}{n^2}$
10	0.01	0.100	1
100	0.0001	0.010	1
1000	0.000001	0.001	1

Almost sure? Add up the chances of a non-zero value: $\sum_{n=1}^{\infty} 1/n^2 = \pi^2/6 \approx 1.645$, a finite total. The first Borel-Cantelli lemma says that when those probabilities sum to something finite, only finitely many of the events happen — so in almost every scenario $X_n$ is eventually zero forever. Yes, $X_n \to 0$ almost surely.

In mean square? Read the last column: $E[X_n^2] = 1$ for every single $n$. It never shrinks. No.

The middle column shows $E[X_n] = 1/n \to 0$, so the sequence does converge in $L^1$ while failing in $L^2$. The culprit is the squaring: a miss of size $n$ arriving with probability $1/n^2$ is negligible on average, but squaring turns $n$ into $n^2$ and exactly cancels the shrinking probability. Rare-but-huge outcomes are invisible to the mean and fatal to the mean square — the same reason a strategy with a fine average return can still break a variance-based risk model.

Worked example 3: distribution convergence proves nothing about closeness

Let $X$ be a standard normal and set $X_n = -X$ for every $n$. The standard normal is symmetric, so $-X$ is also standard normal: every $X_n$ has exactly $X$'s distribution and $X_n \xrightarrow{d} X$ trivially.

But the gap $|X_n - X| = |-X - X| = 2|X|$ does not depend on $n$ at all. Pick $\varepsilon = 1$:

$$P\big( |X_n - X| > 1 \big) = P\big( |X| > 0.5 \big) = 2\big(1 - \Phi(0.5)\big) = 2(1 - 0.6915) = 0.617 .$$

That probability sits at $0.617$ forever, so $X_n$ converges in distribution to $X$ while being, with better-than-even odds, more than a unit away from it. Convergence in distribution compares laws, not values.

What this means in practice

• Which law of large numbers you have. The strong law gives almost sure convergence, so a single long simulation run is guaranteed to settle. The weak law gives only convergence in probability: a long run is probably close, not that a particular run stops wandering. For Monte Carlo you want the strong version, and independent draws with a finite mean give it.
• Consistency versus asymptotic normality. "Consistent" is convergence in probability to the true parameter; "asymptotically normal" is convergence in distribution of the rescaled error. Different claims, different uses.
• Combining limits. Convergence in distribution does not survive addition or multiplication on its own — you need Slutsky's theorem or the continuous mapping theorem.
• Swapping limits and expectations. Convergence in probability does not give $E[X_n] \to E[X]$; worked example 2 is the counterexample. That step needs dominated convergence or uniform integrability.
• Fat tails. Mean square convergence needs finite second moments. Where the variance is questionable, $L^2$ arguments and the standard errors built on them are unavailable, however clean the code looks.