Sensitivity Analysis for Unobserved Confounding

You run a study on historical data and find that funds using a certain risk model earned 2% more per year than funds that didn't. You didn't randomize which funds used the risk model — you just observed who did and who didn't. Maybe the risk model caused the outperformance. Or maybe better-resourced funds both adopt sophisticated risk models and hire better traders, and it's the traders doing the work. You can control for the variables you measured, but you can never rule out a variable you didn't measure. Sensitivity analysis doesn't pretend to solve this — instead it answers a narrower, honest question: how strong would an unmeasured confounder have to be, in order to fully explain away my result?

An analogy: how strong an alibi-breaker would you need?

A detective has a suspect linked to a crime by strong circumstantial evidence. A defense lawyer doesn't need to prove the suspect innocent — they just need to show that a plausible alternative explanation exists. Sensitivity analysis is the detective doing the defense's job in advance: instead of waiting for someone to propose a specific hidden culprit, you calculate how convincing a hidden culprit would need to be — how strongly it would have to be tied to both the "crime" (the outcome) and the "suspect" (the treatment) — to account for all the evidence on its own. If your result only survives against a laughably weak alternative, you should be worried. If it would take an implausibly powerful hidden factor to erase your finding, you can be more confident.

The idea, one symbol at a time

Let $\hat{\tau}$ be your estimated treatment effect (the 2% gap above) and suppose the true effect, after adjusting for an unmeasured confounder $U$, would be smaller or zero. A sensitivity analysis introduces a hypothetical confounder $U$ characterized by two numbers: how strongly $U$ relates to the treatment assignment, and how strongly $U$ relates to the outcome. The E-value approach summarizes both into a single number:

$$E = RR_{obs} + \sqrt{RR_{obs} \times (RR_{obs} - 1)},$$

where $RR_{obs}$ is the observed effect expressed as a risk ratio. In plain English: $E$ is the minimum strength of association — on the risk-ratio scale, applied both to confounder-treatment and confounder-outcome links — that an unmeasured confounder would need to have with both the treatment and the outcome to fully explain away the observed effect. A small $E$ (close to 1) means a weak, entirely plausible confounder could erase your result. A large $E$ means it would take an implausibly strong hidden variable to do so.

Worked example 1: the risk-model study

Suppose the funds using the risk model had an outcome risk ratio of $RR_{obs} = 1.5$ (50% more likely to be top-quartile performers). Plugging in:

$$E = 1.5 + \sqrt{1.5 \times 0.5} = 1.5 + \sqrt{0.75} \approx 1.5 + 0.87 = 2.37.$$

This means an unmeasured confounder — say, "founder pedigree" — would need to be associated with both adopting the risk model and with being top-quartile by a risk ratio of at least 2.37 on each link to fully explain the result. If you believe founder pedigree is at most weakly linked to either (say, a risk ratio of 1.3), it can't fully account for the 2% gap, and your finding survives this particular challenge — though it hasn't been proven causal, only shown robust to this class of alternative story.

Worked example 2: a weaker finding

Now suppose a second study found a smaller gap, $RR_{obs} = 1.1$. Then

$$E = 1.1 + \sqrt{1.1 \times 0.1} = 1.1 + \sqrt{0.11} \approx 1.1 + 0.33 = 1.43.$$

An E-value of 1.43 is much easier to beat — plenty of ordinary confounders (fund size, manager tenure, even which prime broker a fund uses) plausibly have risk-ratio associations above 1.4 with both variables. A weak observed effect needs only a weak hidden confounder to disappear entirely, which is exactly why small effects from observational data deserve more skepticism than large ones.

What this means in practice

Sensitivity analysis is standard practice whenever a result comes from observational data rather than a randomized experiment — factor research, backtest attribution, or any "funds that do X outperform" claim built from historical panel data. Reporting an E-value alongside an effect estimate lets a skeptical reader judge robustness for themselves, rather than trusting a list of controls the author happened to think of. It doesn't replace a randomized experiment (see Randomization Inference in Experiments) — it just quantifies how much an observational claim can bend before it breaks.