Fisher Information

Fisher information answers a very practical question: how much does the data I collected actually pin down the thing I'm trying to estimate? Some experiments are informative, every observation sharply narrows the range of plausible answers. Others are nearly useless, you could collect a thousand points and still barely know the parameter. Fisher information turns that vague sense of "informativeness" into one concrete number.

The idea sits right on top of Maximum Likelihood Estimation (MLE). When you fit a parameter by maximum likelihood, you look for the value $\theta$ that makes the data most probable, and you trace out the log-likelihood $\ell(\theta)$ as $\theta$ varies. If that curve rises to a sharp, pointy peak, tiny changes in $\theta$ hurt the fit a lot, so the data are shouting where the answer is. If the curve is a lazy, flat hump, many values of $\theta$ fit almost equally well, and the data are barely whispering.

The definition

Fisher information is the average curvature of the log-likelihood at the true parameter, equivalently the variance of its slope (the score):

$$I(\theta) = \operatorname{Var}\!\big(\ell'(\theta)\big) = -\,\mathbb{E}\!\left[\ell''(\theta)\right].$$

Reading it in words: $\ell'(\theta)$ is the slope of the log-likelihood (the score), and its variance is how much that slope jumps around from sample to sample. The equivalent form $-\mathbb{E}[\ell''(\theta)]$ is the expected downward curvature, how fast the slope is dropping at the peak. A steep, sharply-bending peak gives a large $I(\theta)$; a flat one gives a small $I(\theta)$. The two expressions being equal is the famous information identity, and it only holds when the model is correctly specified.

Why it matters: it sets the precision floor

The whole reason to care is that Fisher information controls how precise any estimate can possibly be. It adds up across independent observations, $n$ i.i.d. data points carry $n$ times the information of one, $I_n(\theta) = n\,I_1(\theta)$. And through the The Cramér-Rao Bound, the variance of any unbiased estimator can never fall below $1/I_n(\theta)$. More information means a lower floor on your error bars. The maximum-likelihood estimate is special because, in large samples, it reaches that floor: its variance is approximately $1/I_n(\theta)$.

Worked example: information in a coin

You flip a coin $n$ times to estimate the heads probability $p$. For one flip the log-likelihood is $\ell(p) = x\log p + (1-x)\log(1-p)$, where $x$ is $1$ for heads and $0$ for tails. Its second derivative works out to $-x/p^2 - (1-x)/(1-p)^2$, and taking the expectation (using $\mathbb{E}[x] = p$) gives the per-flip information:

$$I_1(p) = \frac{1}{p(1-p)}.$$

Now read what it says. At $p = 0.5$ the information per flip is $1/0.25 = 4$, the smallest it ever gets, so a near-fair coin is the hardest case to pin down. At $p = 0.9$ it jumps to $1/0.09 \approx 11.1$: a lopsided coin reveals itself far faster, because each flip is more decisive. With $n = 100$ flips of a fair coin the total information is $100 \times 4 = 400$, so the best possible variance for estimating $p$ is $1/400 = 0.0025$, a standard error of $0.05$. That matches the familiar $\sqrt{p(1-p)/n} = \sqrt{0.25/100} = 0.05$.

Where it misleads

• It assumes the model is right. The clean identity $I = -\mathbb{E}[\ell'']$ relies on the likelihood being correctly specified. Fit a Gaussian to fat-tailed returns and the "information" you compute overstates your true precision, use a robust (sandwich) covariance instead.
• It's a local, average quantity. Fisher information describes the curvature at the true parameter, averaged over hypothetical datasets. A particular sample can be more or less informative than the average suggests; the observed information (the actual curvature at your estimate) is often the better small-sample proxy.
• High information is not the same as a good answer. Information bounds the variance, not the bias. A confidently precise estimate of the wrong thing is still wrong. See Bias, Variance, and the Quality of Estimators.

The habit worth building: whenever you fit a parameter, glance at how sharply the fit degrades as you move away from the optimum. That steepness is the Fisher information, and it is the raw material behind every standard error, confidence interval, and the efficiency benchmark in the The Cramér-Rao Bound.