Cohen's Kappa and Inter-Rater Agreement

Two research analysts independently label a year of news headlines as "bullish", "bearish", or "neutral" for a sentiment-classification model, and you want to know if they actually agree on what the words mean, or if the model's training labels are just noise. You count how often their labels match: 82%. That sounds solid — until you realize that if most headlines are neutral and both analysts default to "neutral" whenever unsure, two people guessing independently would already agree most of the time by pure chance. Raw agreement can't tell real signal from that background hum. Cohen's kappa fixes this by asking: how much better is the observed agreement than the agreement you'd get if both raters were labeling randomly, given how often each of them uses each label?

An analogy: grading on a curve for chance

Imagine two exam graders marking essays pass/fail, where 90% of essays are obviously passes. Two graders who barely read the essays and just guessed "pass" 90% of the time would still agree with each other about 82% of the time purely by coincidence of both guessing the majority label. If you saw 82% raw agreement between real graders, you'd have no idea whether they were reading carefully or barely trying. Kappa subtracts out that "guessing baseline" so what's left measures actual shared judgment.

The formula, one piece at a time

Let $p_o$ be the observed agreement — the fraction of items where the two raters gave the same label. Let $p_e$ be the expected agreement by chance — the agreement rate you'd predict just from each rater's own label frequencies, if their labels were statistically independent of each other. Cohen's kappa is

$$\kappa = \frac{p_o - p_e}{1 - p_e}.$$

In plain English: kappa measures how much of the possible improvement over chance agreement (the gap between perfect agreement, 1, and chance agreement, $p_e$) was actually achieved. $\kappa = 1$ means perfect agreement, $\kappa = 0$ means the raters do no better than chance, and negative values mean they agree less than chance would predict — a sign something is systematically wrong, like one rater using an inverted convention.

Worked example 1: the sentiment labels

Suppose over 1,000 headlines, analyst A labels 100 bearish, 100 bullish, 800 neutral, and analyst B labels 120 bearish, 80 bullish, 800 neutral, with 820 headlines matching exactly, so $p_o = 820/1000 = 0.82$. Chance agreement is computed label by label from each rater's marginal frequencies: $p_e = (0.10)(0.12) + (0.10)(0.08) + (0.80)(0.80) = 0.012 + 0.008 + 0.64 = 0.66$. Then

$$\kappa = \frac{0.82 - 0.66}{1 - 0.66} = \frac{0.16}{0.34} \approx 0.47.$$

A kappa of 0.47 is only "moderate" agreement by the usual rule of thumb (0.41–0.60), despite 82% raw agreement looking impressive — most of that 82% was the neutral-label chance overlap.

Worked example 2: a rare-event flag

Now suppose two data vendors flag corporate filings as "fraud risk" or not, and fraud flags are rare: vendor A flags 2% of filings, vendor B flags 3%, and they agree on 96.5% of filings overall, so $p_o = 0.965$. Chance agreement: $p_e = (0.02)(0.03) + (0.98)(0.97) = 0.0006 + 0.9506 = 0.9512$. Then $\kappa = (0.965 - 0.9512)/(1 - 0.9512) = 0.0138/0.0488 \approx 0.28$ — "fair" agreement at best, even though 96.5% raw agreement looks close to flawless. With a rare label, near-total agreement is nearly automatic (both vendors saying "no risk" almost always), so kappa is doing real work exposing how little the vendors actually agree on the flags that matter.

What this means in practice

Kappa is the standard check before trusting any hand-labeled dataset used to train or validate a model — sentiment tags, fraud flags, earnings-call topic labels, chart-pattern annotations. If two labelers (or a labeler and a model) show low kappa, the "ground truth" itself is unreliable, and no downstream model can be trusted to exceed the quality of the labels it learned from. It's also used to sanity-check agreement between two data vendors' classifications before merging their feeds, since silently trusting a vendor with 20% hidden disagreement can corrupt a signal without any obvious symptom.