Coupling and Total Variation Distance

How different are two probability distributions? "Different means" is one answer, but crude — two distributions can share a mean and still barely overlap. Total variation distance answers a sharper question: what's the largest possible gap between the probabilities the two distributions assign to any event? Coupling is the constructive trick for computing it: instead of comparing distributions in the abstract, build two random variables on the same coin flips that agree as often as is mathematically possible.

The analogy: two biased dice, matched as often as possible

Imagine two dice with slightly different weightings — die A favors 6s a bit, die B is fair. You want to roll them "together" so that, as often as possible, they land on the same face — using a single shared source of randomness to drive both rolls at once. The best such construction succeeds in matching them with probability exactly $1 - d_{TV}(A,B)$, the total variation distance. Any joint construction — any "coupling" — can do no better; that ceiling is precisely what total variation distance measures.

Writing it down

For distributions $P$ and $Q$ on the same space, total variation distance is

$$d_{TV}(P,Q) = \sup_{A} \big| P(A) - Q(A) \big|,$$

the biggest possible disagreement between $P$ and $Q$ on the probability of any single event $A$. In words: find the event both distributions disagree about the most — that disagreement, at its worst, is the distance. Equivalently, for discrete distributions with probability mass functions $p(x), q(x)$,

$$d_{TV}(P,Q) = \tfrac{1}{2}\sum_x |p(x) - q(x)|.$$

In words: sum up the absolute gap between the two distributions at every outcome, and halve it. A coupling is a joint distribution $(X,Y)$ with $X\sim P$ and $Y \sim Q$ marginally. The coupling inequality says $P(X \ne Y) \ge d_{TV}(P,Q)$ for every coupling, with equality achieved by the best one — called the maximal coupling. In words: no matter how cleverly you try to link two random draws from $P$ and $Q$, they'll disagree at least $d_{TV}(P,Q)$ of the time — and there's always a way to hit that floor exactly.

Compare two normal curves with different means side by side mentally: the region where one curve sits clearly above the other is the "disagreement" total variation distance is built from — shift the mean and watch that overlap shrink.

Worked example 1: two coins

Coin A: $P(\text{heads}) = 0.6$. Coin B: $P(\text{heads})=0.5$. Total variation distance: $d_{TV} = \tfrac12(|0.6-0.5| + |0.4-0.5|) = \tfrac12(0.1+0.1) = 0.1$. So the best possible coupling makes them land the same 90% of the time — construct it explicitly: draw $U \sim \text{Uniform}(0,1)$; set A's result to heads if $U < 0.6$, B's result to heads if $U<0.5$. They agree whenever $U<0.5$ (both heads) or $U\ge0.6$ (both tails) — that's $0.5 + 0.4 = 0.9$ of the time, matching $1-0.1$ exactly.

Worked example 2: Markov chain mixing

A Markov chain modeling a market regime (say, "calm" vs. "stressed" states, each day) has a stationary distribution $\pi$. Starting from "calm" with certainty, after $t$ days the chain's distribution $P_t$ has moved partway to $\pi$. Suppose $d_{TV}(P_t, \pi) = 0.5^t$ (a typical geometric mixing rate for a simple two-state chain with self-transition probability 0.75). After $t=1$: distance $0.5$ — still very far from stationary, the "calm" start dominates. After $t=5$: distance $0.031$ — under 97% coupled with the long-run distribution; for most practical purposes ("has the regime forgotten where it started?") the chain has mixed. This is literally how "mixing time" is defined: the smallest $t$ such that $d_{TV}(P_t,\pi)$ drops below a small threshold like 0.25.

Sample a few paths here and watch how quickly they forget their shared starting point and spread across the range typical of the process — that forgetting is mixing in action, and total variation distance is exactly what measures how much "started here" information is left at any given time.

What this means in practice

• Markov chain Monte Carlo. Total variation distance to the target distribution, tracked via coupling arguments, is the standard way to certify that an MCMC sampler has "burned in" and its samples can be trusted.
• Regime-switching models. How fast a hidden-state model forgets its initial state (calm vs. crisis) is a mixing-time question answered with exactly this machinery.
• Model comparison. Two calibrated models that produce nearly identical means and variances can still have large total variation distance — meaning they disagree sharply on tail events, which is often exactly what matters for risk.