Hit-Rate Inference for Directional Signals

A directional signal says "up" or "down" each day, and over a year of predictions it's right 55% of the time. Fifty-five percent sounds meaningfully better than a coin flip's fifty. But "better than chance" is a statistical question, not a vibe — it depends on exactly how many predictions were made, because a hit rate calculated from a small number of trials is a noisy estimate that can easily land at 55% by pure luck even if the true underlying skill is zero. Hit-rate inference is the standard machinery — built on the binomial distribution — for deciding whether an observed win rate is actually distinguishable from a fair coin.

An analogy: is this coin actually biased?

If you flip a coin 20 times and get 11 heads (55%), nobody would claim the coin is biased — that's an utterly unremarkable outcome for a fair coin, easily explained by ordinary randomness. If you flip the same coin 20,000 times and get 11,000 heads (still exactly 55%), that's an entirely different story — a fair coin landing 55% heads over 20,000 flips is astronomically unlikely, so you'd be quite confident the coin is genuinely biased. The same hit rate, 55%, is either meaningless or overwhelming evidence, purely depending on the sample size. A trading signal's hit rate needs exactly this same scrutiny before anyone calls it skill.

The idea, one symbol at a time

Model each prediction as a Bernoulli trial: correct with true probability $p$, incorrect with probability $1-p$. Under the null hypothesis of no skill, $p = 0.5$. Given $n$ independent predictions with $k$ correct, the observed hit rate is $\hat{p} = k/n$. Its standard error, under the null, is:

$$\text{SE} = \sqrt{\frac{0.5 \times (1-0.5)}{n}} = \frac{0.5}{\sqrt{n}} .$$

In plain English: the uncertainty in a hit-rate estimate shrinks proportionally to $1/\sqrt{n}$ — quadrupling the number of predictions only halves the uncertainty, which is why small samples give such shaky hit-rate estimates. The test statistic for whether $\hat{p}$ is significantly different from 0.5 is:

$$z = \frac{\hat{p} - 0.5}{\text{SE}} = \frac{\hat{p} - 0.5}{0.5/\sqrt{n}} .$$

In plain English: how many standard errors the observed hit rate sits away from the coin-flip benchmark of 50%. Compare $|z|$ to the usual normal critical value (about 1.96 for a 95% two-sided test); if $|z| < 1.96$, the observed hit rate, however good it looks, is not statistically distinguishable from pure chance.

Worked example 1: a modest sample

A signal makes $n = 100$ directional predictions and is correct $k = 58$ times, $\hat{p} = 0.58$.

$$\text{SE} = \frac{0.5}{\sqrt{100}} = 0.05, \qquad z = \frac{0.58 - 0.5}{0.05} = \frac{0.08}{0.05} = 1.6 .$$

$|z| = 1.6 < 1.96$: not statistically significant. A 58% hit rate over just 100 trials is well within the range a coin flip could plausibly produce by chance alone — roughly an 11% probability of seeing a deviation at least this large under the null, not rare enough to be convincing.

Worked example 2: the same hit rate, far more trials

The same signal is instead run for $n = 2{,}500$ predictions, still landing at $\hat{p} = 0.58$ ($k = 1{,}450$).

$$\text{SE} = \frac{0.5}{\sqrt{2500}} = \frac{0.5}{50} = 0.01, \qquad z = \frac{0.08}{0.01} = 8.0 .$$

$|z| = 8.0$ is enormously beyond the 1.96 threshold — with this many trials, a 58% hit rate is essentially impossible to attribute to a fair coin. Identical hit rate, twenty-five times the sample size, completely different conclusion: the first case was inconclusive noise, the second is overwhelming evidence of real predictive skill (or at least, of $p \ne 0.5$; it says nothing yet about whether that edge is large enough to be profitable after costs).

What this means in practice

Hit-rate inference is the quick sanity check for any signal reported as a win percentage — a discretionary trader's "I'm right 60% of the time," a classifier's directional accuracy, or a signal vendor's marketing claim — before treating it as evidence of skill rather than noise. It's also a reminder that a hit rate alone never tells you about profitability: a 51% hit rate on trades with a 3:1 win/loss ratio can be far more profitable than a 65% hit rate on trades with a poor payoff ratio, so hit-rate significance and economic significance are separate questions that both need to be checked.