The Kruskal-Wallis Test

A standard ANOVA compares the means of three or more groups but assumes each group is roughly normally distributed with similar variance — an assumption that breaks down for skewed data like trade P&L or holding periods, which often have long tails and outliers. The Kruskal-Wallis test asks the same question — "do these groups differ?" — without that assumption. It works by pooling all observations from every group together, replacing each raw value with its rank in the combined pool (smallest gets rank 1, and so on), and then checking whether the average rank in each group is close to what you'd expect if group membership were irrelevant. If one group consistently gets much higher or lower ranks than the others, that's evidence the groups come from different distributions.

The test statistic is built from the sum of ranks in each group and, under the null hypothesis that all groups have the same distribution, follows approximately a chi-squared distribution with (number of groups − 1) degrees of freedom. A quant might use it to compare execution slippage across three different brokers, or holding-period returns across four strategy variants, without needing those return distributions to be normal — a real concern given how fat-tailed trading P&L usually is.

A worked feel: with three brokers and 10 trades each, you'd rank all 30 slippage values together, sum the ranks within each broker, and compare that sum to the expected average rank sum of 15.5 per broker (roughly half of 30). Large deviations from 15.5 in any broker's rank sum point to a real difference.