Posterior Predictive Checks

You've fit a Bayesian model to a strategy's daily returns and gotten a clean posterior for its parameters. Everything ran without errors — but does the model actually describe how the strategy behaves? A model can be internally consistent, well-converged, and completely wrong about the real world, for instance assuming returns are symmetric when the strategy has a nasty left tail. Posterior predictive checks catch this: instead of just trusting the fitted parameters, you use the model to generate fake datasets and see whether they resemble the real one.

An analogy: testing a forger by having them draw from memory

Imagine you suspect a forger has learned to reproduce a particular artist's style. The test isn't to ask the forger to describe the style in words — it's to have them produce several new paintings from that learned style, then compare those against genuine originals. If the fakes consistently miss some feature of the originals — say, the brushstroke texture in shadows — you've learned something concrete about how the forger's internal model falls short. A posterior predictive check does the same to a statistical model: generate "fake" datasets from what the model has learned, and compare them to the real data on the features you actually care about.

The maths, one symbol at a time

Let $\theta$ be the model's parameters, $D$ the observed data, and $D^{\text{rep}}$ a "replicated" dataset simulated from the posterior predictive distribution:

$$D^{\text{rep}} \sim p(D^{\text{rep}} \mid D) = \int p(D^{\text{rep}} \mid \theta)\, p(\theta \mid D)\, d\theta$$

In words: draw a parameter value from the posterior, then simulate a full fake dataset as if that parameter value were true, and repeat many times — this is exactly the posterior predictive distribution from before, but now generating whole datasets rather than single points. Pick a test statistic $T(\cdot)$ that captures a feature you care about (e.g. the maximum daily loss, the skewness, the fraction of down-days in a row). Compute $T(D)$ on the real data and compare it to the distribution of $T(D^{\text{rep}})$ across many simulated replicate datasets. If $T(D)$ sits comfortably inside the spread of $T(D^{\text{rep}})$ values, the model captures that feature well; if $T(D)$ is way out in the tail of the simulated distribution, the model is missing something real about the data.

Worked example 1: checking a normal-returns model against real skew

A model assumes a strategy's daily returns are normally distributed, and its posterior is well-fit. Simulate 1,000 replicate datasets of 250 daily returns each and compute the skewness of each. Suppose the simulated skewness values center near $0$, with 90% between $-0.3$ and $0.3$ (expected — a normal model can't generate much skew). The real data's skewness turns out to be $-1.4$ — far outside that range. This is a clear check failure: the normal model, however well its mean and variance fit, cannot reproduce the real returns' sharp left tail, and any risk number derived from it will understate tail risk.

Worked example 2: checking a fitted win-rate model against streakiness

A model assumes each trade's win/loss is an independent Bernoulli draw with posterior win rate around $0.55$. Simulate 500 replicate sequences of 100 trades and compute the longest losing streak in each. Suppose simulated streaks average 5–6 trades, with 95% between 3 and 9, but the real history shows a losing streak of 15 in a row. That value falls far outside the simulated range, flagging that the independence assumption is likely wrong — real losses are probably clustering, which an i.i.d. Bernoulli model cannot produce no matter how the win-rate parameter is tuned.

Think of the explorer above as generating a batch of simulated samples the way a posterior predictive check does — the question is always whether the real, single observed value looks like a typical draw from that simulated batch, or like an outlier.

What this means in practice

Posterior predictive checks are the Bayesian equivalent of residual diagnostics — they catch model misspecification that a well-converged posterior alone won't reveal, since a model can fit the mean and variance of the data well while still being wrong about tail behavior, streakiness, or seasonality. Picking the right test statistics is the actual skill: check features you specifically care about downstream (tail losses if you're sizing risk, streak lengths if you're setting stop-loss rules), not just generic goodness-of-fit numbers.