Privacy and Memorisation in Synthetic Data

"Synthetic" sounds like it should mean "safe" — no real record is being handed out, only a model's imagined approximation of one. That intuition is wrong more often than people expect. A generative model with enough capacity relative to its training set can, in effect, memorise individual training examples and later reproduce them almost exactly, especially the rarest or most distinctive ones. A dataset built this way isn't a genuinely new sample from the underlying distribution — it's partially a shuffled re-release of the original records, with the privacy problem intact.

The analogy: a student who memorised the answer key

A student who genuinely understands the material can solve new problems they've never seen. A student who only memorised the answer key can reproduce answers to problems they've seen before, word for word, but is exposed the moment a genuinely new problem appears — and, worse, if you ask them to "make up a similar problem," they might hand back one of the exact problems from the key instead of inventing something new. A generative model that has memorised training examples is the second student: fluent-looking output that is, on closer inspection, just a copy of something it was shown.

How memorisation is actually detected

The standard check is a nearest-neighbor distance test: for each generated (synthetic) sample, find its closest match in the real training set (by some distance measure appropriate to the data — Euclidean distance on returns, edit distance on sequences, etc.), and compare that distance to the typical distance between two different real training points. If synthetic samples sit implausibly close to specific real records — much closer than real records typically sit to each other — that's evidence of memorisation rather than genuine generalisation.

$$d_{\min}(\hat{x}) = \min_{x \in \mathcal{D}_{\text{train}}} \lVert \hat{x} - x \rVert$$

In words: for a synthetic sample $\hat{x}$, find the single closest real training example and measure the distance to it; a suspiciously small $d_{\min}(\hat{x})$ compared to typical real-to-real distances flags a likely memorised copy.

Worked example 1: a numeric memorisation flag

Suppose the median nearest-neighbor distance between real training records themselves is $4.2$ (in whatever units the feature space uses — say, normalized return-path distance). For 1,000 synthetic samples, the median nearest-real-neighbor distance comes out to $4.0$ — close to the real-to-real baseline, a healthy sign of genuine generalisation. But the minimum across those 1,000, i.e. the single closest synthetic-to-real match, is $0.03$ — two orders of magnitude closer than typical. That one sample is very likely a near-exact copy of a specific real record, even though the aggregate (median) statistic looked fine; aggregate closeness checks can hide a handful of severe individual leaks.

Worked example 2: rare events are disproportionately at risk

A synthetic financial-scenario generator trained on 10 years of daily data has roughly 2,500 training days, of which only a handful — say 5 — are extreme crash days. A model with enough capacity can find it "cheaper," in likelihood terms, to memorise those 5 rare, hard-to-model days outright rather than learn a general rule for tail behavior, since 5 specific memorised points cost little capacity and get the loss on them to near zero. Practically: if a synthetic dataset's very worst days match specific real historic crash dates almost exactly (correlation $>0.99$ with a real historic path, say), that's a strong sign the tails were memorised rather than modeled — precisely the region where privacy leakage (revealing which real firm or fund experienced which exact drawdown) matters most.

Push the correlation this high and imagine each pair being one real point and its nearest synthetic "neighbor" — a near-perfect match like this, repeated for specific individual points rather than the dataset overall, is exactly the memorisation signature the nearest-neighbor test is built to catch.

What this means in practice

Anywhere synthetic data is meant to substitute for sensitive originals — client trade histories, proprietary strategy P&L paths, or individual account records used to train a scenario generator — a memorisation audit is a required step before release, not an optional extra. Techniques like differential privacy (bounding how much any single training record can influence the model) and explicit rare-event down-weighting are used to reduce the risk, at some cost to how well genuinely rare, important patterns get learned at all.