Fisher's Exact Test

The chi-square test of independence for a 2x2 table relies on a large-sample approximation — it treats the test statistic as following a chi-square distribution, which is only accurate when every cell in the table has a reasonably large expected count (a common rule of thumb is at least 5). When sample sizes are small — say, testing whether a rare trading signal fired more often before big moves than before ordinary days, with only a handful of big-move events — that approximation breaks down and the chi-square test's p-values become unreliable.

Fisher's exact test avoids the approximation entirely. Given a 2x2 table with fixed row and column totals, it computes the exact probability of observing a table as extreme or more extreme than the one seen, directly from the hypergeometric distribution, rather than approximating with a continuous distribution.

$$p = \frac{\binom{a+b}{a}\binom{c+d}{c}}{\binom{n}{a+c}}$$

In words: given the row and column totals are fixed, this is the exact probability of the specific cell counts $a, b, c, d$ occurring by chance, computed by counting how many ways the totals could be arranged versus how many arrangements are as extreme as what was observed.

For example, suppose a signal fired on 3 of 10 "big move" days and 1 of 20 "ordinary" days. With such small counts, chi-square's approximation is shaky, but Fisher's exact test can still compute a valid p-value directly from the hypergeometric probabilities of that exact table and every more extreme table with the same margins.