The Kolmogorov-Smirnov Test

You've fit a normal distribution to a year of daily returns and want to know: does the data actually look normal, or is that assumption going to bite you in the tails? Or you've backtested two versions of a strategy and want to know if their trade-return distributions genuinely differ, not just their means. Both questions are really the same question in disguise: are two distributions — one empirical, one theoretical or empirical — the same, or meaningfully different? The Kolmogorov-Smirnov (KS) test compares two entire cumulative distribution curves at once, rather than reducing each to a single summary number like a mean or variance first.

An analogy: overlaying two staircases

Picture the empirical distribution function (see The Empirical Distribution Function) of your data as a staircase, and the theoretical distribution you're testing against as a smooth curve — or, for a two-sample test, a second staircase from a different dataset. Lay one directly on top of the other and find the single point where they're vertically furthest apart. If that biggest gap is small, the two curves hug each other closely everywhere, meaning the distributions are similar across their entire range, not just on average. If the gap is large, somewhere along the range one distribution has noticeably more (or less) mass below a point than the other — and the test turns "how large is too large" into a precise probability statement.

The idea, one symbol at a time

The KS statistic for comparing an empirical CDF $\hat{F}_n(x)$ to a reference CDF $F_0(x)$ is

$$D_n = \sup_x \left| \hat{F}_n(x) - F_0(x) \right|,$$

the largest absolute vertical distance between the two curves over all $x$. In plain English: check every possible threshold $x$, measure how far off your empirical distribution is from the reference distribution at that threshold, and $D_n$ is the single worst-case gap. For two samples, the two-sample version replaces $F_0$ with a second empirical CDF $\hat{F}_m$. Under the null hypothesis that the sample really is drawn from $F_0$ (or that both samples share a common distribution), $D_n$ has a known distribution — and critically, one that does not depend on the shape of $F_0$ itself (a property called distribution-free), which lets a single table of critical values work for testing against any reference distribution, normal, exponential, or otherwise.

Worked example 1: a five-point normality check by hand

Suppose 5 sorted, standardized daily returns (mean 0, sd 1): $-1.8, -0.4, 0.1, 0.6, 1.3$. The ECDF steps up at $0.2, 0.4, 0.6, 0.8, 1.0$. Comparing to the standard normal CDF $\Phi(x)$ at each point, checking both just-before and at each jump (since the largest gap can occur on either side): at $x=-1.8$, $\Phi(-1.8)\approx0.036$, and $\hat{F}_5$ jumps from 0 to 0.2 there, giving gaps $0.036$ and $0.164$. Checking the remaining points similarly and taking the largest gap overall gives, say, $D_5 \approx 0.20$ (illustrative). With only 5 points, the 5% critical value is around 0.56, so $D_5=0.20$ is far from significant — you'd need a much bigger gap, or far more data, to reject normality.

Worked example 2: two-sample comparison of strategy returns

Now compare 100 trade returns from Strategy A against 100 from Strategy B. Suppose their empirical CDFs are close everywhere except around the median, where Strategy A's CDF sits noticeably higher (meaning A has more small/negative outcomes concentrated there) — giving a two-sample $D_{100,100} = 0.18$. The approximate critical value at the 5% level for two samples of size 100 each is roughly $1.36\sqrt{\frac{1}{100}+\frac{1}{100}} \approx 0.192$. With $D=0.18$ just under that threshold, the test narrowly fails to reject the null of equal distributions at 5% — a close call that would flip with a slightly larger sample or a slightly bigger gap, illustrating how KS test conclusions can be sensitive right around a real difference of this size.

What this means in practice

The KS test is a standard goodness-of-fit check whenever you've assumed a distribution shape — testing whether return residuals really look normal before relying on formulas that assume normality, or comparing two backtests' full return distributions rather than just their Sharpe ratios. Its distribution-free property makes it convenient, but it is a blunt instrument for tail behavior specifically, since the CDFs of most distributions are close together in the tails almost by construction (both curves are near 0 or near 1 there).