Order Statistics

Take a random sample of $n$ numbers and sort them from smallest to largest. The sorted values are the order statistics. We write the smallest as $X_{(1)}$, the next as $X_{(2)}$, and so on up to the largest, $X_{(n)}$. The minimum, the maximum, the median, the 90th percentile of a sample — these are all order statistics, and they answer the questions that plain averages can't: how bad is the worst case, how long until the first failure, how big is the record so far.

The reason order statistics need their own theory is that even when your raw draws are independent and identically distributed, the sorted values are neither: the minimum and maximum are tied together by the very act of sorting. So we build their distributions from scratch.

The two easy ones: max and min

The maximum is small only if every draw is small. Since the draws are independent, the probability they're all below a level $x$ is the product of the individual probabilities. If each draw has cumulative distribution $F(x) = P(X \le x)$, then

$$P(X_{(n)} \le x) = F(x)^n .$$

The minimum works the mirror-image way: the minimum exceeds $x$ only if every draw exceeds $x$, so

$$P(X_{(1)} > x) = \bigl(1 - F(x)\bigr)^n .$$

Both come straight from "all of them at once = multiply." The general $k$-th order statistic has a tidier-looking density built from the same logic, but the max and min carry most of the intuition.

Worked example: the expected max of five uniform draws

Suppose you draw five independent numbers uniformly from $0$ to $1$. Where do you expect the largest to land? A beautiful result says that for the uniform on $0$ to $1$, the $k$-th smallest of $n$ draws has expected value

$$E\bigl[X_{(k)}\bigr] = \frac{k}{n+1} .$$

The $n$ draws chop the interval into $n+1$ gaps that are equal on average, so the sorted values sit at the fence-posts. With $n = 5$: the expected minimum is $\tfrac{1}{6} \approx 0.167$, the expected median is $\tfrac{3}{6} = 0.5$, and the expected maximum is $\tfrac{5}{6} \approx 0.833$. Notice the maximum crawls toward $1$ only slowly — even with $n = 5$ it's not at $0.99$, it's at $0.83$. Adding more draws pushes it up as $\tfrac{n}{n+1}$, a stubbornly gradual climb.

Worked example: first failure among many

Order statistics of the exponential have a clean shortcut. If $n$ components each fail at an independent exponential time, the minimum — the first failure — is itself exponential, but with the rates added together. Say you run $10$ machines and each has a mean life of $100$ days (rate $\tfrac{1}{100}$ per day). The first failure among the ten is exponential with rate $10 \times \tfrac{1}{100} = \tfrac{1}{10}$, so its mean is

$$E\bigl[\text{first failure}\bigr] = \frac{1}{10} \text{ per day} \;\Rightarrow\; 10 \text{ days}.$$

Ten machines fail first ten times sooner than one machine fails on its own. This is exactly why systems built from many parts break far earlier than any single part's average life suggests.

Pitfalls

• Extremes are volatile. The maximum and minimum have much larger sampling variability than the mean; a single new record can be far from the last one, so never treat an observed max as a stable estimate.
• The tail shape is everything. The behaviour of $X_{(n)}$ is driven entirely by the far tail of $F$. Heavy tails make the maximum wander enormously, which is why fat-tailed markets produce record moves that a normal model would call impossible.
• Independence and identical distribution are assumed. Sorting correlated or non-identical draws breaks the clean product formulas above.