Importance Sampling

Plain Monte Carlo works by sampling from the true distribution and averaging. That is fine when the thing you want happens often. But suppose you are estimating the probability of a rare event — a 1-in-30,000 tail loss. Sample naively and almost every draw lands nowhere near the tail, contributing a useless zero, and you would need hundreds of thousands of samples just to see the event a handful of times. Importance sampling fixes this by cheating cleverly: draw samples from a different distribution that spends its time where the action is, then correct for the swap so the answer stays exactly right.

The reweighting identity

Say you want the average of $f(X)$ when $X$ follows the true density $p$. The trick is to multiply and divide by a second density $q$ that you get to choose:

$$E_p[f(X)] = E_q\!\left[f(X)\,\frac{p(X)}{q(X)}\right].$$

Read this carefully: instead of averaging $f$ over draws from $p$, you average $f$ times a weight $w(X) = p(X)/q(X)$ over draws from $q$. The weight — the ratio of how likely the sample was under the truth versus under your sampler — undoes the bias you introduced by sampling from the wrong place. The estimate becomes

$$E_p[f(X)] \approx \frac{1}{N}\sum_{i=1}^{N} f(X_i)\,\frac{p(X_i)}{q(X_i)}, \qquad X_i \sim q.$$

Choose $q$ to put its mass where $f \cdot p$ is large — the region that actually contributes to the answer — and each sample now carries real information instead of a zero.

Worked example: a deep tail probability

You want $P(X > 4)$ for a standard normal $X$ — the chance of a four-sigma move. The true answer is about $3.17 \times 10^{-5}$, roughly one in 31,500. Sample naively from $N(0,1)$ and you would draw around 31,500 numbers just to expect a single hit; the estimate would be hopelessly noisy at any practical sample size.

Instead, sample from a proposal shifted right onto the tail: $q = N(4, 1)$. Now suppose one draw comes out at $x = 4.2$ — squarely in the region of interest, so $f(x) = 1$. Its weight is the ratio of the two densities:

$$w = \frac{p(4.2)}{q(4.2)} = \exp\!\Big(-\tfrac{4.2^2}{2} + \tfrac{(4.2 - 4)^2}{2}\Big) = \exp\!\big(-8.8\big) \approx 1.5 \times 10^{-4}.$$

Every draw from $q$ now lands in the tail, and each contributes a small weight around $10^{-4}$; average those weights and you converge on the tiny true probability with a few thousand samples instead of millions. Same answer, a fraction of the compute.

Where it goes wrong

The whole method lives or dies on the choice of $q$, and a bad choice is worse than doing nothing.

• Thin-tailed proposals explode. If $q$ has lighter tails than $p$, then out in the tail the weight $p/q$ blows up, a handful of freak samples dominate the average, and the estimator's variance can become infinite — you get a confident-looking number that is silently garbage. The safe rule: $q$ should have tails at least as fat as $p$ wherever $f$ is nonzero.
• Coverage is non-negotiable. $q$ must be able to reach everywhere $p \cdot f$ is nonzero. If $q$ assigns zero probability to a region that actually matters, that region is invisible and the estimate is biased.
• Weight degeneracy. In high dimensions the weights often become wildly unequal — one sample carries almost all the weight and the rest count for nothing, collapsing your effective sample size. Diagnosing this (via the effective sample size) is part of using the method responsibly.

Importance sampling is how practitioners estimate rare-but-costly quantities: deep-tail value-at-risk and expected shortfall, the price of far-out-of-the-money options, and the loss distributions behind tail-risk hedges — anywhere the events that matter most are the ones you almost never sample.