Kolmogorov's Axioms of Probability

Before 1933, "probability" meant different things to different mathematicians, a long-run frequency to some, a degree of belief to others, and there was no single set of rules everyone agreed a valid probability had to obey. Andrey Kolmogorov fixed this with three simple rules. Everything else in probability theory, Bayes' rule, expectation, the central limit theorem, is a consequence of just these three, so it is worth seeing exactly what they say and why nothing less would do.

The analogy: rules for a fair coin jar

Imagine a jar holding tickets for every possible outcome of some experiment, every card in the filing cabinet from set theory. You want to assign each pile of tickets (each event) a "size" that behaves the way sizes should: nothing is negatively sized, the whole jar has a fixed total size, and if you split the jar into non-overlapping piles, their sizes must add up to the size of the piles combined. Kolmogorov's axioms say exactly this, with "size" renamed "probability."

Writing it down

Fix a sample space $\Omega$ (every possible outcome) and consider events $A \subseteq \Omega$ (sub-piles of outcomes, in the language of sets). A function $P$ assigning a number to every event is a valid probability measure if it satisfies three axioms.

Axiom 1, non-negativity. $P(A) \geq 0$ for every event $A$. In words: no probability is ever negative; the smallest a pile can be sized is zero.

Axiom 2, normalisation. $P(\Omega) = 1$. In words: something in the sample space is guaranteed to happen, so the whole jar has total size exactly one.

Axiom 3, countable additivity. For any sequence of pairwise disjoint events $A_1, A_2, A_3, \dots$ (no two overlap),

In words: if you split an event into non-overlapping pieces, even infinitely many of them, the probability of the whole thing is the sum of the probabilities of the pieces, and no more. This is the direct probabilistic cousin of the set-theory fact that disjoint piles don't need an overlap correction.

Everything familiar follows from just these three. For instance $P(A^c) = 1 - P(A)$ falls straight out: $A$ and $A^c$ are disjoint and together make up $\Omega$, so by Axiom 3 their probabilities sum to $P(\Omega) = 1$ by Axiom 2.

Worked example 1: a loaded six-sided die

Suppose a die is weighted so that $P(1) = P(2) = 0.05$ each, $P(3)=P(4)=P(5)=0.1$ each, and $P(6) = 0.6$. Check the axioms directly: each value is $\geq 0$ (Axiom 1, satisfied). Sum them: $0.05+0.05+0.1+0.1+0.1+0.6 = 1.00$ (Axiom 2, satisfied, since these six outcomes are disjoint and exhaust $\Omega$). Now compute $P(\text{even})$ using Axiom 3: the events "roll 2," "roll 4," "roll 6" are pairwise disjoint, so $P(\text{even}) = P(2)+P(4)+P(6) = 0.05+0.1+0.6 = 0.75$. You never needed anything beyond the three axioms to get that number.

Worked example 2: proving the union bound from the axioms

Claim: for any two events, $P(A \cup B) \leq P(A) + P(B)$. Split $A \cup B$ into two disjoint pieces: $A$ itself, and "$B$ but not $A$," written $B \setminus A$. These are disjoint by construction, and together they make up $A \cup B$, so Axiom 3 gives $P(A \cup B) = P(A) + P(B \setminus A)$. Now, $B \setminus A$ is a sub-pile of $B$, and a sub-pile can never have more probability than the pile containing it (this itself follows from Axiom 1 applied to $B \setminus (B\setminus A)$), so $P(B \setminus A) \leq P(B)$. Substituting: $P(A \cup B) = P(A) + P(B\setminus A) \leq P(A) + P(B)$. The whole inequality, used constantly to bound the chance that at least one of many bad things happens, is a two-line consequence of the axioms, not a separate assumption.

What this means in practice

Every probability model a quant builds, a distribution of returns, a default probability, a Bayesian posterior, is required to satisfy these three axioms or it is not a valid probability model at all, regardless of how it was derived. Risk aggregation, VaR bounds, and union-bound arguments in machine learning generalisation theory all trace back to Axiom 3.