The Hill Estimator and Tail Index

"How fat is the tail?" sounds like a question that needs a whole distribution fit first. It doesn't. If you only care about the far tail, the part that decides whether a 10-sigma move is merely rare or genuinely impossible, there's a single number, the tail index, that summarizes it, and a formula that reads it straight off your largest observations without assuming anything about the rest of the distribution.

The analogy: judging a staircase by its top steps

A power-law tail means that, plotted on a log-log scale, the far right of the distribution looks like a straight line, its steepness is the tail index. You don't need to survey the whole staircase to measure that steepness; you can estimate the slope just from the last few steps at the top. The Hill estimator does exactly that: it looks only at the $k$ largest observations in your sample and estimates the tail's steepness from how they're spaced relative to each other.

Writing it down

A distribution has a power-law (Pareto-type) tail with index $\alpha$ if $P(X > x) \sim C x^{-\alpha}$ for large $x$, smaller $\alpha$ means a fatter, slower-decaying tail (more extreme events), and $\alpha \le 2$ means the variance itself is infinite. Sort your sample from largest to smallest as $X_{(1)} \ge X_{(2)} \ge \cdots$, pick a number of top observations $k$, and the Hill estimator is

In words: take the $k$ largest observations, compute the log-ratio of each to the $(k{+}1)$-th largest (the cutoff just below your chosen tail), average those log-ratios, and invert. If the top observations are only slightly larger than the cutoff, the average log-ratio is small and $\hat\alpha$ comes out large, a thin tail. If some of the top observations are wildly larger than the cutoff, the average log-ratio is large and $\hat\alpha$ comes out small, a fat tail.

The explorer above plots a power curve; a Pareto-type tail's survival probability $P(X>x)$ decays exactly like this shape on the right side of the distribution, and $\alpha$ is the exponent controlling how sharply it bends toward zero, drag it and notice how a smaller exponent leaves far more mass out in the tail.

Worked example 1: computing $\hat\alpha$ by hand

Suppose the six largest daily losses in a sample (in $ millions), sorted descending, are 12, 9, 8, 7, 6, 5, and you take $k=5$, using the 6th value (5) as the cutoff. Compute the log-ratios: $\ln(12/5)=0.875$, $\ln(9/5)=0.588$, $\ln(8/5)=0.470$, $\ln(7/5)=0.336$, $\ln(6/5)=0.182$. Sum: $0.875+0.588+0.470+0.336+0.182 = 2.451$. Average: $2.451/5 = 0.490$. Invert: $\hat\alpha = 1/0.490 \approx 2.04$. A tail index just above 2 is a classic finding for daily equity return tails, right at the boundary where variance is barely finite, which is exactly the empirical folklore this estimator is built to quantify precisely.

Worked example 2: why $k$ matters, on the same data

Redo it with only the top $k=3$ observations (12, 9, 8), cutoff now the 4th value, 7: $\ln(12/7)=0.539$, $\ln(9/7)=0.251$, $\ln(8/7)=0.134$. Sum $=0.924$, average $=0.308$, so $\hat\alpha = 1/0.308 \approx 3.25$. Using fewer, more extreme observations gave a different answer, 3.25 instead of 2.04, from the same six numbers. This isn't a mistake in either calculation; it's the estimator's defining weakness: with so few data points feeding it, $\hat\alpha$ is highly sensitive to exactly how many top observations you include, and there's no automatic right answer for $k$.

What this means in practice

• Tail index below 2 means theoretically infinite variance, a warning that any variance-based risk measure (standard deviation, ordinary VaR formulas that assume finite variance) is standing on ground that doesn't fully exist for that instrument's return distribution.
• Cross-asset comparisons of "fat-tailedness", equities versus crypto versus commodities, are routinely done by comparing Hill estimates, since it needs no full distributional assumption, only enough extreme observations to work with.
• A "Hill plot" (plotting $\hat\alpha$ against a range of $k$ values) is the standard diagnostic for whether an estimate is trustworthy: a genuine power-law tail shows $\hat\alpha$ roughly flat over a wide range of $k$; a jagged, unstable Hill plot means the tail isn't well-described by a single power law, or there isn't enough data.