Bayes' Theorem

Bayes' theorem is how rational belief updates on evidence. It looks like a one-line algebraic rearrangement of conditional probability, and it is, but its consequences are counterintuitive enough that it fools trained physicians, and its structure is general enough that it underlies signal filtering, spam detection, credit scoring, and every model that "learns" from data. The whole discipline is contained in one move: turn the conditional you can measure into the conditional you actually want.

From conditional probability to the theorem

Conditional probability is defined on a Probability Spaces by $\mathbb{P}(A \mid B) = \mathbb{P}(A \cap B)/\mathbb{P}(B)$ for $\mathbb{P}(B) > 0$. Writing the joint probability two ways, $\mathbb{P}(A\cap B) = \mathbb{P}(A\mid B)\mathbb{P}(B) = \mathbb{P}(B\mid A)\mathbb{P}(A)$, and solving gives Bayes' theorem: $\mathbb{P}(A \mid B) = \frac{\mathbb{P}(B \mid A)\,\mathbb{P}(A)}{\mathbb{P}(B)}.$ In inference language, with hypothesis $H$ and evidence $E$: $\underbrace{\mathbb{P}(H \mid E)}_{\text{posterior}} = \frac{\overbrace{\mathbb{P}(E \mid H)}^{\text{likelihood}}\ \overbrace{\mathbb{P}(H)}^{\text{prior}}}{\underbrace{\mathbb{P}(E)}_{\text{evidence}}}.$ The denominator is the normaliser, expanded by the law of total probability over a partition $\{H_i\}$: $\mathbb{P}(E) = \sum_i \mathbb{P}(E \mid H_i)\,\mathbb{P}(H_i).$ The single most important structural fact: the posterior is proportional to likelihood times prior, $\mathbb{P}(H\mid E) \propto \mathbb{P}(E\mid H)\,\mathbb{P}(H)$, with the denominator merely rescaling to sum to one. Everything hard in Bayesian computation is that normaliser; everything conceptual is the proportionality.

The base-rate trap

Bayes' theorem's most famous lesson is that the prior dominates when the base rate is extreme, and human intuition systematically ignores it. The canonical worked example:

A disease affects $0.1\%$ of the population. A test has $99\%$ sensitivity ($\mathbb{P}(+\mid D) = 0.99$) and $99\%$ specificity ($\mathbb{P}(-\mid D^c) = 0.99$, so the false-positive rate is $0.01$). You test positive. What is $\mathbb{P}(D \mid +)$? $\mathbb{P}(D\mid +) = \frac{0.99 \cdot 0.001}{0.99\cdot 0.001 + 0.01\cdot 0.999} = \frac{0.00099}{0.00099 + 0.00999} = \frac{0.00099}{0.01098} \approx 9.0\%.$ Despite a "99% accurate" test, a positive result means only a 9% chance of disease. The reason is the base rate: healthy people are 1000× more numerous, so their $1\%$ false-positive stream ($\approx 9.99$ per 1000) swamps the $0.99$ true positives. The intuition-shattering point, that the answer depends as much on the prior $0.001$ as on the test's accuracy, is the trap. In trading terms: a signal with a great hit rate on a rare event still generates mostly false alarms, and sizing to the raw hit rate over-bets.

The odds / log-odds form

For sequential updating, the ratio form is far cleaner. Dividing Bayes for $H$ by Bayes for $H^c$ cancels the intractable $\mathbb{P}(E)$: $\underbrace{\frac{\mathbb{P}(H\mid E)}{\mathbb{P}(H^c\mid E)}}_{\text{posterior odds}} = \underbrace{\frac{\mathbb{P}(E\mid H)}{\mathbb{P}(E\mid H^c)}}_{\text{likelihood ratio}}\times \underbrace{\frac{\mathbb{P}(H)}{\mathbb{P}(H^c)}}_{\text{prior odds}}.$ Taking logs turns multiplication into addition, so each piece of independent evidence adds a fixed number of "bits" of log-odds to your belief: $\log\text{-odds}_{\text{post}} = \log\text{-odds}_{\text{prior}} + \sum_j \log\frac{\mathbb{P}(E_j\mid H)}{\mathbb{P}(E_j\mid H^c)}.$ This is exactly the logistic-regression / naive-Bayes classifier, and it is how belief revision is implemented in practice: accumulate evidence as additive log-likelihood ratios. It also makes the base-rate lesson vivid, a strong prior is just a large starting log-odds that weak evidence cannot overcome.

Conjugacy: keeping the posterior tractable

The general posterior requires integrating $\mathbb{P}(E)$, often impossible in closed form. A conjugate prior is one that, combined with a given likelihood, yields a posterior in the same family, so updating is just parameter arithmetic. The workhorse example is the Beta–Binomial: model an unknown success probability $\theta$ (a fill rate, a win rate) with prior $\theta \sim \text{Beta}(\alpha, \beta)$. Observe $k$ successes in $n$ trials (Binomial likelihood). Because the posterior is proportional to $\theta^{k}(1-\theta)^{n-k}\cdot \theta^{\alpha-1}(1-\theta)^{\beta-1}$, $\theta \mid \text{data} \sim \text{Beta}(\alpha + k,\ \beta + n - k).$ You literally add successes to $\alpha$ and failures to $\beta$. The prior parameters act as pseudo-counts, $\text{Beta}(1,1)$ is a uniform "no information" prior, and larger $\alpha+\beta$ is a more confident prior needing more data to move. Other standard conjugate pairs: Normal–Normal (a Gaussian prior on a mean with Gaussian likelihood stays Gaussian, the basis of the Kalman filter update) and Gamma–Poisson (for rates). Conjugacy is why simple Bayesian models run in real time.

Failure modes and subtleties

• Ignoring the base rate. The dominant real-world error, as above: a positive test / firing signal on a rare event is mostly false positives. Always ask "how rare is the thing?" before trusting a conditional.
• Garbage priors. Bayes is only as good as the prior; a badly wrong, overconfident prior (small variance, far from truth) can take enormous data to overturn. "Let the data speak" fails when the prior is stubborn.
• The likelihood must be the true generating model. A misspecified $\mathbb{P}(E\mid H)$ produces a confidently wrong posterior; Bayesian updating does not detect model error, it propagates it.
• Independence abuse (naive Bayes). The additive log-odds form assumes evidence pieces are conditionally independent given $H$. When they are correlated (many signals reading the same underlying factor), you double-count and become overconfident, a common trap in stacking "independent" alpha signals.
• $\mathbb{P}(E) = 0$ evidence. Conditioning on a measure-zero event needs care (regular conditional distributions); naive plug-in can give paradoxes (Borel–Kolmogorov).

In interviews

The base-rate / medical-test question is a near-certainty on any probability screen, practise producing the ~9% answer fast and explaining why it is so far below 99%. Expect the odds form ("update your belief after two independent positive tests") where multiplying likelihood ratios is the slick path. Be ready to state conjugacy for Beta–Binomial and interpret the prior as pseudo-counts. A deeper prompt is "when does more data not fix a bad prior?", near-degenerate priors or misspecified likelihoods. Bayesian updating over states is the engine of belief propagation in Markov Chains and filtering, so a strong answer connects the one-shot rule to sequential estimation.