The Hill Estimator and Tail Index

"How fat is the tail?" sounds like a question that needs a whole distribution fit first. It doesn't. If you only care about the far tail — the part that decides whether a 10-sigma move is merely rare or genuinely impossible — there's a single number, the tail index, that summarizes it, and a formula that reads it straight off your largest observations without assuming anything about the rest of the distribution.

The analogy: judging a staircase by its top steps

A power-law tail means that, plotted on a log-log scale, the far right of the distribution looks like a straight line — its steepness is the tail index. You don't need to survey the whole staircase to measure that steepness; you can estimate the slope just from the last few steps at the top. The Hill estimator does exactly that: it looks only at the $k$ largest observations in your sample and estimates the tail's steepness from how they're spaced relative to each other.

Writing it down

A distribution has a power-law (Pareto-type) tail with index $\alpha$ if $P(X > x) \sim C x^{-\alpha}$ for large $x$ — smaller $\alpha$ means a fatter, slower-decaying tail (more extreme events), and $\alpha \le 2$ means the variance itself is infinite. Sort your sample from largest to smallest as $X_{(1)} \ge X_{(2)} \ge \cdots$, pick a number of top observations $k$, and the Hill estimator is

$$\hat{\alpha} = \left( \frac{1}{k} \sum_{i=1}^{k} \ln\frac{X_{(i)}}{X_{(k+1)}} \right)^{-1}.$$

In words: take the $k$ largest observations, compute the log-ratio of each to the $(k{+}1)$-th largest (the cutoff just below your chosen tail), average those log-ratios, and invert. If the top observations are only slightly larger than the cutoff, the average log-ratio is small and $\hat\alpha$ comes out large — a thin tail. If some of the top observations are wildly larger than the cutoff, the average log-ratio is large and $\hat\alpha$ comes out small — a fat tail.

The explorer above plots a power curve; a Pareto-type tail's survival probability $P(X>x)$ decays exactly like this shape on the right side of the distribution, and $\alpha$ is the exponent controlling how sharply it bends toward zero — drag it and notice how a smaller exponent leaves far more mass out in the tail.

Worked example 1: computing $\hat\alpha$ by hand

Suppose the six largest daily losses in a sample (in $ millions), sorted descending, are 12, 9, 8, 7, 6, 5, and you take $k=5$, using the 6th value (5) as the cutoff. Compute the log-ratios: $\ln(12/5)=0.875$, $\ln(9/5)=0.588$, $\ln(8/5)=0.470$, $\ln(7/5)=0.336$, $\ln(6/5)=0.182$. Sum: $0.875+0.588+0.470+0.336+0.182 = 2.451$. Average: $2.451/5 = 0.490$. Invert: $\hat\alpha = 1/0.490 \approx 2.04$. A tail index just above 2 is a classic finding for daily equity return tails — right at the boundary where variance is barely finite, which is exactly the empirical folklore this estimator is built to quantify precisely.

Worked example 2: why $k$ matters, on the same data

Redo it with only the top $k=3$ observations (12, 9, 8), cutoff now the 4th value, 7: $\ln(12/7)=0.539$, $\ln(9/7)=0.251$, $\ln(8/7)=0.134$. Sum $=0.924$, average $=0.308$, so $\hat\alpha = 1/0.308 \approx 3.25$. Using fewer, more extreme observations gave a different answer — 3.25 instead of 2.04 — from the same six numbers. This isn't a mistake in either calculation; it's the estimator's defining weakness: with so few data points feeding it, $\hat\alpha$ is highly sensitive to exactly how many top observations you include, and there's no automatic right answer for $k$.

What this means in practice

• Tail index below 2 means theoretically infinite variance — a warning that any variance-based risk measure (standard deviation, ordinary VaR formulas that assume finite variance) is standing on ground that doesn't fully exist for that instrument's return distribution.
• Cross-asset comparisons of "fat-tailedness" — equities versus crypto versus commodities — are routinely done by comparing Hill estimates, since it needs no full distributional assumption, only enough extreme observations to work with.
• A "Hill plot" (plotting $\hat\alpha$ against a range of $k$ values) is the standard diagnostic for whether an estimate is trustworthy: a genuine power-law tail shows $\hat\alpha$ roughly flat over a wide range of $k$; a jagged, unstable Hill plot means the tail isn't well-described by a single power law, or there isn't enough data.