Maximum A Posteriori Estimation

Maximum likelihood estimation picks whichever parameter value makes the observed data most probable, full stop — it has no memory of anything you believed before you saw the data. That is a problem with small samples: three coin flips that all come up heads gives a maximum-likelihood estimate of "this coin always lands heads," which is an absurd conclusion for what is very likely a fair coin that got a short unlucky (or lucky) streak. Maximum a posteriori (MAP) estimation fixes this by folding in a prior belief about the parameter, then picking the value that is most probable given both the prior and the data together.

The analogy: a detective with a hunch and new evidence

A detective walks into a case with some prior sense of how likely different suspects are, based on experience — most burglaries are committed by someone with a record, not a random stranger. New evidence arrives — a fingerprint, a partial description — and the detective doesn't throw out their prior hunch and start from zero; they combine the hunch with the new evidence to land on the single most probable suspect given everything known so far, prior hunch included. MAP estimation is that combination made precise: it picks the parameter value most probable after multiplying "how likely is this parameter value a priori" by "how well does this parameter value explain the data I actually observed."

The formula: prior times likelihood, maximized

Bayes' rule says the posterior probability of a parameter $\theta$ given data $D$ is proportional to the likelihood times the prior:

$$\hat\theta_{\text{MAP}} = \arg\max_\theta \; p(D \mid \theta)\, p(\theta)$$

In words: instead of maximizing just the likelihood $p(D\mid\theta)$ (how well $\theta$ explains the data, which is what maximum likelihood does alone), MAP multiplies that likelihood by the prior $p(\theta)$ (how plausible $\theta$ was believed to be before seeing any data) and finds the $\theta$ that makes this product largest. The normalizing constant from Bayes' rule, $p(D)$, doesn't depend on $\theta$, so it can be dropped entirely when just searching for the maximizing value.

Worked example 1: the three-heads coin, MLE versus MAP

Three flips, all heads. Maximum likelihood picks $\hat p_{\text{MLE}} = 1.0$ (the fraction of heads observed) because $p(D\mid p) = p^3$ is maximized at $p=1$. Now add a prior that most coins are close to fair: a Beta$(10,10)$ prior, which is symmetric and concentrated around $p=0.5$. The posterior for a Beta prior combined with binomial data is another Beta distribution, Beta$(10+3, 10+0) =$ Beta$(13,10)$, and the mode (the MAP estimate) of a Beta$(\alpha,\beta)$ distribution is $\frac{\alpha-1}{\alpha+\beta-2}$:

$$\hat p_{\text{MAP}} = \frac{13-1}{13+10-2} = \frac{12}{21} \approx 0.571$$

Three heads pulled the estimate only from $0.5$ to about $0.571$, not all the way to $1.0$ — the strong prior belief that coins are usually close to fair absorbed most of the shock of a short, unlucky-for-fairness streak.

Worked example 2: what happens with more data

Same setup, but now 30 flips, 27 heads. Maximum likelihood: $\hat p_{\text{MLE}} = 27/30 = 0.9$. With the same Beta$(10,10)$ prior, the posterior is Beta$(10+27, 10+3) =$ Beta$(37,13)$, and the MAP estimate is:

$$\hat p_{\text{MAP}} = \frac{37-1}{37+13-2} = \frac{36}{48} = 0.75$$

With more data, MAP moved much closer to the data-driven estimate ($0.75$ versus MLE's $0.9$, compared to the 3-flip case where MAP stayed near $0.57$ versus MLE's $1.0$) — the prior's influence shrinks automatically as evidence accumulates, exactly as it should: a strong prior should matter most when data is scarce, and matter less and less as data piles up and can speak for itself.

Drag the prior and watch the posterior above update as data comes in — the peak of that posterior curve is precisely the MAP estimate: the single most probable parameter value once prior and data are combined.

What this means in practice

MAP estimation is the practical bridge between full Bayesian inference (which returns an entire posterior distribution, often expensive to compute) and plain maximum likelihood (which returns a single point estimate but ignores prior knowledge entirely): MAP returns a single point estimate too, but one regularized by a prior. In fact, common machine learning regularizers are MAP estimates in disguise — L2 regularization (ridge regression) is exactly MAP estimation under a Gaussian prior on the coefficients, and L1 regularization (lasso) is MAP under a Laplace prior; the "regularization strength" hyperparameter is really controlling how tight that implicit prior is.

Practice

1. Recompute the MAP estimate for the coin example with a much stronger prior, Beta$(100,100)$, using the 3-heads data, and comment on how much less the estimate moves.
2. Explain in one sentence why the MAP estimate approaches the MLE estimate as the amount of data grows, regardless of the prior chosen.
3. Ridge regression is MAP estimation under what kind of prior on the regression coefficients?