Evaluating Synthetic Market Data

A generative model trained on historical prices can produce a chart that, glanced at, looks exactly like a real one — jagged, trending, plausible. That is a dangerously low bar. Real markets have specific, well-documented statistical fingerprints — returns that are far more extreme than a normal distribution would predict, volatility that clusters into calm and turbulent periods, and near-zero correlation between today's return and tomorrow's — and a generative model can fail every one of these tests while still producing a chart that fools the eye. If synthetic data is going to be used to stress-test a strategy or augment a small training set, it has to be checked against these fingerprints quantitatively, not visually.

The analogy: a forger's brushstrokes vs the chemistry of the paint

An art forger can copy a painting's visible brushstrokes well enough to fool a casual viewer. What actually catches forgeries is analyzing things invisible to the eye — the chemical composition of the pigment, the age of the canvas fibers, the layering order of the paint. A synthetic price series is the same problem in reverse: the "brushstrokes" are the visible up-and-down wiggle of the chart, easy to fake, and the "chemistry" is a set of statistical properties — fat tails, volatility clustering, the absence of easy autocorrelation to exploit — that a naive generator gets wrong even while nailing the visible wiggle. Evaluating synthetic data means running the chemistry tests, not admiring the brushstrokes.

The mechanism: comparing distributions, not single numbers

The core statistical tool is comparing the distribution of some statistic computed on real data against the same statistic computed on synthetic data, rather than comparing single summary numbers like the mean. One common comparison uses the Kullback-Leibler divergence between a real distribution $P$ and a synthetic distribution $Q$:

$$D_{\text{KL}}(P \,\|\, Q) = \sum_{i} P(i) \log \frac{P(i)}{Q(i)}$$

Here $i$ indexes possible outcomes (for instance, buckets of daily return sizes), $P(i)$ is the fraction of real days falling in bucket $i$, and $Q(i)$ is the fraction of synthetic days falling in the same bucket. In words: for every bucket, weight the log-ratio of real-versus-synthetic frequency by how common that bucket actually is in the real data, and sum. A value of exactly zero means the two distributions match perfectly bucket for bucket; larger values mean the synthetic data's shape diverges more from reality, and the divergence is especially punished in buckets where real data is common but synthetic data is rare.

Three specific "stylized facts" get checked this way in practice: the tail heaviness of the return distribution (do 3-standard-deviation moves happen about as often in synthetic data as in real data, or far less often?), volatility clustering (does a big move today make a big move tomorrow more likely, in both real and synthetic series, measured by autocorrelation of squared or absolute returns?), and near-zero return autocorrelation (real markets show almost no predictability from yesterday's raw return to today's — if synthetic data shows strong, easily-tradable autocorrelation, it has invented an arbitrage that doesn't exist in reality, which is exactly the opposite of realistic).

Worked example 1: comparing tail frequency by hand

Over 1,000 real trading days, suppose 8 days had a return move beyond 3 standard deviations — a rate of $8/1{,}000 = 0.8\%$. A normal distribution would predict about $0.27\%$ (roughly 2.7 per 1,000), so real markets already show fatter tails than normal. Now generate 1,000 synthetic days from a model and count only 2 such extreme days, a rate of $0.2\%$. The synthetic generator has understated tail risk by a factor of 4 relative to real data ($0.8\%$ versus $0.2\%$) — a strategy backtested only on this synthetic data would see far fewer stress days than reality delivers, and any risk limit calibrated on it would be dangerously loose. This single frequency comparison, computed with nothing more than a count and a ratio, would have caught a failure invisible on a price chart.

Worked example 2: checking volatility clustering with autocorrelation

Take the absolute daily returns (ignoring sign, just size of move) for 6 real days: $1.0\%, 1.2\%, 0.3\%, 0.2\%, 1.5\%, 1.4\%$ — notice the big moves cluster (days 1–2, then days 5–6), a signature of real volatility clustering. The correlation between $|\text{return}_t|$ and $|\text{return}_{t+1}|$ across these adjacent pairs is positive: big-move days tend to follow big-move days. Now suppose a synthetic generator, sampling each day independently, produces the same six magnitudes but in a random order that happens to give $0.3\%, 1.4\%, 1.0\%, 0.2\%, 1.5\%, 1.2\%$ — same set of numbers, same mean, same standard deviation, but no adjacent-day clustering at all, because each day was drawn independently with no memory of the previous day. A model can match every single-day statistic perfectly and still fail this test, because clustering is a property of pairs of days, not of any one day in isolation.

What this means in practice

Synthetic market data gets used to stress-test strategies against scenarios rarer than the historical record contains, and to augment small training sets for models that need more examples than history provides. Both uses are actively harmful if the synthetic data understates tail risk or invents autocorrelation that doesn't exist — a strategy that looks profitable on synthetic data with a fake exploitable pattern will look far worse, or fail outright, on real markets once deployed. Evaluation against stylized facts — tail frequency, volatility clustering, near-zero raw-return autocorrelation — is not optional polish; it is the check that determines whether the synthetic data is usable at all.

Practice

1. A synthetic generator produces daily returns with the correct mean and standard deviation but draws each day fully independently. Which stylized fact will it necessarily fail, and why does independence guarantee that failure?
2. Real data shows 3-sigma moves at 4× the normal-distribution rate; a generator shows them at 3.5× the normal-distribution rate but with the wrong sign skew (too many large up-moves relative to down-moves). Design one more numeric check beyond raw frequency that would catch this.
3. Why is comparing histograms of returns, bucket by bucket, a stronger test than comparing just the mean and standard deviation of the two datasets?