Kolmogorov's Axioms of Probability

Before 1933, "probability" meant different things to different mathematicians — a long-run frequency to some, a degree of belief to others — and there was no single set of rules everyone agreed a valid probability had to obey. Andrey Kolmogorov fixed this with three simple rules. Everything else in probability theory — Bayes' rule, expectation, the central limit theorem — is a consequence of just these three, so it is worth seeing exactly what they say and why nothing less would do.

The analogy: rules for a fair coin jar

Imagine a jar holding tickets for every possible outcome of some experiment — every card in the filing cabinet from set theory. You want to assign each pile of tickets (each event) a "size" that behaves the way sizes should: nothing is negatively sized, the whole jar has a fixed total size, and if you split the jar into non-overlapping piles, their sizes must add up to the size of the piles combined. Kolmogorov's axioms say exactly this, with "size" renamed "probability."

Writing it down

Fix a sample space $\Omega$ (every possible outcome) and consider events $A \subseteq \Omega$ (sub-piles of outcomes, in the language of sets). A function $P$ assigning a number to every event is a valid probability measure if it satisfies three axioms.

Axiom 1 — non-negativity. $P(A) \geq 0$ for every event $A$. In words: no probability is ever negative; the smallest a pile can be sized is zero.

Axiom 2 — normalisation. $P(\Omega) = 1$. In words: something in the sample space is guaranteed to happen, so the whole jar has total size exactly one.

Axiom 3 — countable additivity. For any sequence of pairwise disjoint events $A_1, A_2, A_3, \dots$ (no two overlap),

$$P\left(\bigcup_{i=1}^{\infty} A_i\right) = \sum_{i=1}^{\infty} P(A_i) .$$

In words: if you split an event into non-overlapping pieces — even infinitely many of them — the probability of the whole thing is the sum of the probabilities of the pieces, and no more. This is the direct probabilistic cousin of the set-theory fact that disjoint piles don't need an overlap correction.

Everything familiar follows from just these three. For instance $P(A^c) = 1 - P(A)$ falls straight out: $A$ and $A^c$ are disjoint and together make up $\Omega$, so by Axiom 3 their probabilities sum to $P(\Omega) = 1$ by Axiom 2.

Worked example 1: a loaded six-sided die

Suppose a die is weighted so that $P(1) = P(2) = 0.05$ each, $P(3)=P(4)=P(5)=0.1$ each, and $P(6) = 0.6$. Check the axioms directly: each value is $\geq 0$ (Axiom 1, satisfied). Sum them: $0.05+0.05+0.1+0.1+0.1+0.6 = 1.00$ (Axiom 2, satisfied, since these six outcomes are disjoint and exhaust $\Omega$). Now compute $P(\text{even})$ using Axiom 3: the events "roll 2," "roll 4," "roll 6" are pairwise disjoint, so $P(\text{even}) = P(2)+P(4)+P(6) = 0.05+0.1+0.6 = 0.75$. You never needed anything beyond the three axioms to get that number.

Worked example 2: proving the union bound from the axioms

Claim: for any two events, $P(A \cup B) \leq P(A) + P(B)$. Split $A \cup B$ into two disjoint pieces: $A$ itself, and "$B$ but not $A$," written $B \setminus A$. These are disjoint by construction, and together they make up $A \cup B$, so Axiom 3 gives $P(A \cup B) = P(A) + P(B \setminus A)$. Now, $B \setminus A$ is a sub-pile of $B$, and a sub-pile can never have more probability than the pile containing it (this itself follows from Axiom 1 applied to $B \setminus (B\setminus A)$), so $P(B \setminus A) \leq P(B)$. Substituting: $P(A \cup B) = P(A) + P(B\setminus A) \leq P(A) + P(B)$. The whole inequality — used constantly to bound the chance that at least one of many bad things happens — is a two-line consequence of the axioms, not a separate assumption.

What this means in practice

Every probability model a quant builds — a distribution of returns, a default probability, a Bayesian posterior — is required to satisfy these three axioms or it is not a valid probability model at all, regardless of how it was derived. Risk aggregation, VaR bounds, and union-bound arguments in machine learning generalisation theory all trace back to Axiom 3.