Analysis of Variance and F-Tests

Comparing two trading strategies, a t-test does the job directly. Comparing five strategies at once, the obvious instinct — run every pairwise t-test — quietly breaks the statistics: with five groups there are ten pairs, and running ten tests each at a 5% false-alarm rate stacks up a much higher combined chance of a fluke "winner," the exact family-wise error problem. Analysis of variance (ANOVA) sidesteps this by asking one single, honest question first: do these groups differ at all? Only if the answer is yes does it make sense to dig into which ones.

An analogy: are these classrooms actually different, or just noisy?

Five classrooms take the same exam. Their average scores differ — some by a little, some by a lot — but so does the spread of scores within each individual classroom, student to student. If the gap between classroom averages is small compared to how much scores naturally bounce around within any one classroom, the differences between classrooms are probably just noise. If the gap between classroom averages is large compared to the within-classroom spread, something real is likely distinguishing the classrooms — a better teacher, a different curriculum. ANOVA is exactly this ratio, formalized: variation between groups relative to variation within groups.

The formula, one symbol at a time

Split the total variability in the data into two pieces: between-group variance (how much the group means differ from the overall average) and within-group variance (how much individual observations differ from their own group's mean). The F-statistic is their ratio:

$$F = \frac{\text{variance between groups}}{\text{variance within groups}} = \frac{MSB}{MSW}$$

In plain English: $MSB$ (mean square between) measures how spread out the group averages are from each other; $MSW$ (mean square within) measures how noisy each group is internally. If the groups are all really drawn from populations with the same true mean, $F$ should hover around 1 — between-group spread and within-group spread should look similar, since both are just reflecting the same underlying noise. A large $F$, well above 1, means the groups are more spread apart from each other than ordinary within-group noise would explain. $F$ is compared against an F-distribution with $(k-1, N-k)$ degrees of freedom, where $k$ is the number of groups and $N$ the total observations.

$MSB$ and $MSW$ are each built from squared, roughly-normal deviations like the ones the curve above describes — $MSB$ from how far group means sit from the grand mean, $MSW$ from how far individual points sit from their own group mean — and the F-distribution captures the shape of their ratio.

Worked example 1: three strategies, is there a real difference?

Three strategies, 10 return observations each. Their group means: $0.10\%, 0.08\%, 0.15\%$; grand mean across all 30 observations: $0.11\%$. Suppose the computed sums of squares give $MSB = 0.0018$ and $MSW = 0.0012$ (in squared-percent units).

$$F = \frac{MSB}{MSW} = \frac{0.0018}{0.0012} = 1.50$$

With $k-1=2$ and $N-k=27$ degrees of freedom, the critical value at $\alpha=0.05$ is about $3.35$. Since $1.50 < 3.35$, there is not enough evidence that the three strategies' true means actually differ — the gaps between $0.10\%$, $0.08\%$, and $0.15\%$ are consistent with ordinary within-strategy noise.

Worked example 2: a clearer separation

Same setup, but now suppose the between-group spread is larger relative to noise: $MSB = 0.0060$, $MSW = 0.0010$.

$$F = \frac{0.0060}{0.0010} = 6.00$$

$6.00$ exceeds the critical value of $3.35$ for $(2, 27)$ degrees of freedom — this result is statistically significant. The conclusion at this stage is only that at least one of the three strategies' true means differs from the others; ANOVA's F-test does not by itself say which one, or by how much — that requires a follow-up (a post-hoc test, itself typically run with a multiple-testing correction such as Bonferroni or Holm).

What this means in practice

ANOVA is the right first step whenever comparing more than two groups at once — multiple strategy variants, multiple parameter settings in a grid search, multiple market regimes' average returns — before drilling into pairwise comparisons. Running it first, and only proceeding to pairwise tests if the F-test is significant, is a disciplined way to avoid exactly the fishing-expedition problem that plagues naive "just t-test every pair" approaches to comparing several backtested variants.