Bayesian Networks and Joint Factorization

The joint probability of 10 binary variables, written out in full, has $2^{10} - 1 = 1{,}023$ independent numbers to specify — and that count doubles with every additional variable. No one can write down, estimate, or reason about a table that size directly. A Bayesian network is the standard fix: draw the variables as nodes in a directed graph, with an arrow from each variable to whatever it directly influences, and that graph turns out to be exactly the information needed to factor the enormous joint distribution into a handful of small, local pieces — one modest table per node, instead of one astronomical table for everything at once.

The analogy: a recipe versus a finished dish's full ingredient interactions

Describing every possible flavor interaction between 10 ingredients in a dish directly — how salt interacts with garlic interacts with heat interacts with acid, all at once — would take an enormous, unmanageable table. A recipe never does this. It instead lists a sequence of small local steps: "sauté the garlic in oil" depends only on garlic and oil, not on what happens six steps later. The full flavor of the dish emerges from chaining these small, local dependencies together, not from ever writing down one giant combined table. A Bayesian network is this recipe structure applied to probability: each variable's distribution is specified only in terms of its direct parents in the graph, not the entire rest of the network, and the full joint distribution is recovered by chaining these local pieces together.

The factorization formula

A Bayesian network over variables $X_1, \dots, X_n$ with graph structure $G$ factors the joint distribution as:

$$p(X_1, \dots, X_n) = \prod_{i=1}^{n} p\big(X_i \mid \text{parents}(X_i)\big)$$

In words: the probability of everything happening together is just the product, over every variable, of that variable's probability given only its direct parents in the graph — no variable's local table needs to mention any node it isn't directly connected to. If a variable has no parents, its local table is just its plain marginal probability.

Worked example 1: a 3-node chain, factoring the joint by hand

Take three variables: $R$ (heavy rain today, yes/no), $T$ (train delayed, yes/no), $L$ (arrives late to work, yes/no), with the causal chain $R \to T \to L$ (rain delays the train, a delayed train causes lateness — $L$ has no direct dependence on $R$ except through $T$). The full joint would naively need $p(R,T,L)$ specified for all 8 combinations. The graph says instead: $p(R,T,L) = p(R)\,p(T\mid R)\,p(L\mid T)$. Suppose $p(R=\text{yes})=0.2$, $p(T=\text{yes}\mid R=\text{yes})=0.7$, $p(T=\text{yes}\mid R=\text{no})=0.1$, $p(L=\text{yes}\mid T=\text{yes})=0.8$, $p(L=\text{yes}\mid T=\text{no})=0.05$. The probability of "it rained, the train was delayed, and I was late" is: $p(R=\text{yes})\,p(T=\text{yes}\mid R=\text{yes})\,p(L=\text{yes}\mid T=\text{yes}) = 0.2 \times 0.7 \times 0.8 = 0.112$. Only three small numbers multiplied together, none of which ever required a table over all three variables jointly.

Worked example 2: counting the savings

The chain above needs: 1 number for $p(R)$, 2 numbers for $p(T\mid R)$ (one per value of $R$), 2 numbers for $p(L\mid T)$ — 5 numbers total (the complementary probabilities like $p(R=\text{no})$ are implied by subtracting from 1). The naive full joint table over 3 binary variables needs $2^3-1 = 7$ independent numbers — already a saving, and the gap widens fast. Scale to 10 binary variables arranged as a chain (each depending only on the one before it): the network needs $1 + 9\times 2 = 19$ numbers total, versus $2^{10}-1 = 1{,}023$ for the full joint table — a roughly 50-fold reduction, purely from the graph telling you which variables can be ignored when specifying each local piece.

The exploding curve above is the same shape as the naive joint table's size, $2^n - 1$, as the number of variables $n$ grows — the whole reason Bayesian network factorization, which grows only linearly in a chain, matters at all.

What this means in practice

This factorization is what makes probabilistic reasoning over many variables computationally tractable at all — it underlies everything from medical diagnosis systems (symptom nodes depending on disease nodes) to credit risk models (default depending on a handful of directly relevant factors, not every economic variable simultaneously) to the structure behind causal DAGs used for identifying confounders. The graph itself is a modeling choice, usually built from domain knowledge about what genuinely causes or influences what — a wrong graph (missing an arrow, or drawing one that shouldn't exist) doesn't just make the model less accurate, it changes which conditional independencies the model assumes are true, which is exactly the subject of d-separation.

Practice

1. For the rain-train-late chain, compute $p(R=\text{no}, T=\text{yes}, L=\text{no})$ using the same numbers given.
2. If a 4th variable, "meeting missed," depended only on $L$, write out the new factorized joint distribution formula.
3. Explain in one sentence why a Bayesian network's factorization only works correctly if the graph has no directed cycles.