The Chi-Square Goodness-of-Fit Test

Roll a die 60 times and a perfectly fair die won't land exactly 10 times on each face — some faces will come up 8 times, others 13, purely by chance. The question worth asking isn't "did every face land exactly 10 times?" (it almost never will) but "are these deviations from 10 the kind of wobble a fair die naturally produces, or so large that the die is probably loaded?" The chi-square goodness-of-fit test is the tool that turns that question into a number, and it generalizes far past dice — to whether a market's order sizes follow a theorized distribution, or whether a credit portfolio's rating buckets match a benchmark mix.

An analogy: a bakery's daily orders versus its forecast

A bakery predicts it will sell 20% croissants, 30% muffins, 30% bagels, and 20% donuts each day. On a given day with 100 orders, it actually sells 18 croissants, 35 muffins, 28 bagels, 19 donuts. None of these match the forecast exactly — should the bakery panic and rebuild its forecasting model, or is this just an ordinary day's random fluctuation? The chi-square test answers by asking: for each category, how far off was the actual count from the forecast, relative to how large a deviation you'd naturally expect just from day-to-day randomness with these totals?

The formula, one symbol at a time

Let $O_i$ be the observed count in category $i$, and $E_i$ be the count the theoretical model expected. The chi-square statistic sums, across all $k$ categories, the squared gap relative to expectation:

$$\chi^2 = \sum_{i=1}^{k} \frac{(O_i - E_i)^2}{E_i}$$

In plain English: for each category, square how far off the actual count was from the expected count, then scale that squared gap by the expected count itself, and add all of these up. Dividing by $E_i$ matters because a gap of 5 is a big deal in a category expected to have 10 observations, but trivial in one expected to have 1,000. The resulting $\chi^2$ value is compared against a chi-square distribution with $k-1$ degrees of freedom (one fewer than the number of categories, because the counts must sum to the total, removing one degree of freedom) — a large $\chi^2$ relative to that reference distribution means the observed counts are unlikely to have come from the theorized model.

A chi-square statistic is, loosely, built from squaring and summing several roughly-normal deviations like the one pictured above — which is exactly why it only takes positive values and why it has its own distinctive right-skewed shape rather than the familiar symmetric bell curve.

Worked example 1: is the die fair?

Sixty rolls of a die, expected count $E_i = 10$ for each of the six faces (a uniform $1/6$ each), observed counts: $8, 13, 9, 11, 7, 12$.

$$\chi^2 = \frac{(8-10)^2}{10} + \frac{(13-10)^2}{10} + \frac{(9-10)^2}{10} + \frac{(11-10)^2}{10} + \frac{(7-10)^2}{10} + \frac{(12-10)^2}{10}$$ $$\chi^2 = \frac{4}{10}+\frac{9}{10}+\frac{1}{10}+\frac{1}{10}+\frac{9}{10}+\frac{4}{10} = \frac{28}{10} = 2.8$$

With $k-1 = 5$ degrees of freedom, the critical value at $\alpha=0.05$ is $11.07$. Since $2.8 < 11.07$, there is no evidence the die is unfair — these deviations are entirely ordinary noise for 60 rolls of a fair die.

Worked example 2: the bakery's day

Expected counts from the forecast (out of 100 orders): croissants $20$, muffins $30$, bagels $30$, donuts $20$. Observed: $18, 35, 28, 19$.

$$\chi^2 = \frac{(18-20)^2}{20}+\frac{(35-30)^2}{30}+\frac{(28-30)^2}{30}+\frac{(19-20)^2}{20}$$ $$\chi^2 = \frac{4}{20}+\frac{25}{30}+\frac{4}{30}+\frac{1}{20} = 0.2+0.833+0.133+0.05 = 1.217$$

With $k-1=3$ degrees of freedom, the critical value at $\alpha=0.05$ is $7.81$. Since $1.217$ is far below $7.81$, this day's order mix is entirely consistent with the bakery's forecast — no need to rebuild the model over one ordinary day.

What this means in practice

The test is a standard first check on whether an empirical distribution matches a theoretical one — validating whether trade sizes bucket into the frequencies a market microstructure model predicts, whether a scoring model's predicted rating buckets match realized default-rate buckets, or whether a random-number generator used in a simulation is actually producing uniform draws. It is also a useful sanity check on classification models: bucketing predicted probabilities and comparing them to realized outcome frequencies is a chi-square test in disguise, closely related to the calibration checks used alongside The Confusion Matrix and Classification Metrics.