Influence Functions and the Breakdown Point

Two estimators can agree perfectly on clean data and disagree completely on how they'd behave if the data got dirty. You need a way to grade "how fragile is this estimator" before the bad data shows up, not after your number has already been ruined. Statisticians have two such grades: one asks how far a single rogue point can push the estimate, and the other asks how many rogue points it takes to break the estimate entirely.

An analogy: a seesaw and its tipping point

Think of a seesaw loaded with people of different weights sitting at different distances from the center — that is your dataset, and the balance point is your estimator. Now imagine one person on the seesaw can walk anywhere they like, including far off to one end. Two questions matter: first, how much does the balance point move per step that person takes — that is the influence function. Second, if you're allowed to replace some fraction of the people with new, arbitrarily heavy people placed wherever you want, what's the smallest fraction that lets you drag the balance point anywhere at all — that is the breakdown point.

The rule, one symbol at a time

Let $T$ be your estimator (a formula that turns data into a number) and let $F$ be the true distribution the clean data comes from. Contaminate that distribution by adding an infinitesimal speck of extra probability, weight $\varepsilon$, sitting exactly at a point $x$. Call the contaminated distribution $F_\varepsilon = (1-\varepsilon)F + \varepsilon \delta_x$, where $\delta_x$ just means "a spike of probability at $x$." The influence function is

$$\text{IF}(x; T, F) = \lim_{\varepsilon \to 0} \frac{T(F_\varepsilon) - T(F)}{\varepsilon} .$$

In plain English: it's the rate of change of the estimate as you nudge in an infinitesimal speck of contamination at the point $x$ — the slope of "estimate versus location of one extra bad point." For the mean, $\text{IF}(x) = x - \mu$, which grows without limit as $x$ moves to infinity: one point far enough away can drag the mean anywhere. For the median, the influence function is bounded — it caps out at a constant regardless of how far $x$ goes, which is exactly the flat line in the picture.

The breakdown point $\varepsilon^*$ is a cruder, more practical number: the smallest fraction of the data you'd need to replace with arbitrarily bad values before the estimator can be forced to any answer at all (including infinity). In plain English: the percentage of the data an attacker would need to corrupt to make your estimator say anything they want.

Picture one extra point landing ever further into the tail of the curve above. An unbounded influence function lets that single point pull the estimate arbitrarily far as it moves out; a bounded one caps its pull at a fixed ceiling no matter how far out it lands.

Worked example 1: influence function of the mean, by hand

Take five points: $2, 3, 4, 5, 6$, so $\mu = 4$. Add a sixth point at $x = 100$ with tiny weight $\varepsilon = 0.01$ (treat it as one extra observation among 100 effective units). The new mean is approximately $(1-\varepsilon)\cdot 4 + \varepsilon \cdot 100 = 3.96 + 1.00 = 4.96$, a jump of $0.96$ for $\varepsilon = 0.01$, i.e. a slope of about $0.96/0.01 = 96$. Compare to the formula $\text{IF}(x) = x - \mu = 100 - 4 = 96$. Matches exactly. Now push $x$ to $1{,}000$: $\text{IF} = 996$ — ten times farther out gives ten times the damage, with no ceiling.

Worked example 2: breakdown point of the mean vs. the median

With $n = 10$ clean points, how many can an attacker corrupt to send the mean to $+\infty$? Just one — replace a single point with an arbitrarily large number and the sum, hence the mean, diverges. The mean's breakdown point is $1/10 = 10\%$, worsening toward $1/n \to 0$ as $n$ grows. For the median of the same 10 points, an attacker needs roughly half the data — at least 5 of 10 points — to move it arbitrarily far, because the median only looks at the middle-ranked value and ignores how extreme corrupted points individually are. Its breakdown point is close to $50\%$, the best possible for any reasonable estimator.

What this means in practice

• Choosing an estimator for dirty data. Anything with a breakdown point near $1/n$ — the plain mean, least-squares regression, sample covariance — is only as trustworthy as your data-cleaning pipeline. One bad tick can move it without limit.
• Huber estimators sit in between. Bounded influence like the median, but a breakdown point below 50% — a deliberate trade-off for keeping more efficiency on clean data.
• Risk models. Sample covariance matrices have poor breakdown points; a handful of corrupted returns can distort every correlation the optimizer relies on.