Always-Valid Inference and e-Values

A trader running a live A/B test between two execution algorithms wants to know as soon as possible which one is better, and checks the results every morning. An ordinary p-value is only trustworthy if you fix the sample size in advance and look exactly once. Check it every day and stop the first morning it dips below 0.05, and the true chance of a false alarm is no longer 5% — it can climb past 30% or 50%, because you gave yourself many chances to get lucky. Always-valid inference, built on a tool called the e-value, is designed to survive exactly this kind of repeated peeking: you can look after every single trade and stop whenever you want, and the guarantee against false alarms still holds.

An analogy: betting against the null hypothesis

Imagine you're allowed to bet, at odds you set yourself, against the idea that "nothing is different" (the null hypothesis). Before seeing any data, you commit to fair-odds wagers: if the next observation looks like what the null predicts, you lose a bit of your stake; if it looks like the alternative, you win a bit. Your running wealth, after any number of bets, is the e-value. Because the bets were fair under the null, your wealth has no tendency to drift up if the null is actually true — on average it just holds steady. A basic fact about such fair betting games (Ville's inequality) says the chance your wealth ever multiplies itself by 20 or more, at any point during an unboundedly long sequence of fair bets, is at most 1/20. That single fact is what lets you monitor continuously: no matter when you stop and look, the odds of having been fooled are capped.

The mechanics, one symbol at a time

Let $e$ denote the e-value after some number of observations — think of it as your current wealth in the betting game, starting at $e = 1$. A valid e-value must satisfy $\mathbb{E}_0[e] \le 1$, where $\mathbb{E}_0$ is the expectation if the null hypothesis is really true — in words, if nothing is going on, your wealth shouldn't grow on average. You build $e$ by multiplying together a small "bet factor" for each new observation: a factor bigger than 1 if the observation favors the alternative, smaller than 1 if it favors the null. You reject the null the moment $e$ crosses a threshold $1/\alpha$ (for a 5% test, that's $1/0.05 = 20$), and this stopping rule is valid regardless of when you decide to stop — you can check after every trade, every hour, or only on Fridays, and the false-alarm rate stays capped at $\alpha$.

Worked example 1: building the wealth process by hand

Suppose the null hypothesis is that an algorithm fills 50% of its passive orders (a coin-flip). Design a fair bet: multiply the running e-value by 1.4 on a fill and by 0.6 on a miss. This is mean-preserving under the null because $0.5 \times 1.4 + 0.5 \times 0.6 = 1$, so $\mathbb{E}_0[e] = 1$ exactly, as required. After 10 trades with 7 fills and 3 misses:

$$e = 1.4^{7} \times 0.6^{3} \approx 10.54 \times 0.216 \approx 2.28 .$$

That's evidence in favor of the alternative (fills are happening more than 50% of the time), but it's nowhere near the reject threshold of 20, so you keep collecting data — and because this is an e-value, you're allowed to.

Worked example 2: crossing the threshold mid-stream

The algorithm keeps running at the same true 70% fill rate. After 40 trades total (28 fills, 12 misses):

$$e = 1.4^{28} \times 0.6^{12} \approx 12{,}342 \times 0.00218 \approx 26.9 .$$

That's above $1/\alpha = 20$, so you reject the null right there, at trade 40 — even though you also checked and failed to reject at trade 10, and at every trade in between. Ordinary p-values don't allow this: recomputing a fresh p-value at every trade and stopping the first time it looks significant would have inflated the real false-positive rate well above 5%. The e-value's multiply-as-you-go construction is exactly what keeps the guarantee intact under continuous peeking.

What this means in practice

Always-valid inference is what makes it safe to run live dashboards on trading experiments — A/B tests of execution algorithms, signal changes, or risk overlays — and to stop a test as soon as the evidence is clear, rather than pre-committing to a fixed sample size and staring at a locked dashboard until it's reached. It's the modern engine behind sequential monitoring tools, and it composes cleanly: e-values from independent tests can simply be multiplied together, which makes combining evidence across desks or time periods far easier than combining ordinary p-values correctly.