Multi-Label Classification

Ordinary multiclass classification assumes every example gets exactly one label — a bond is investment-grade or high-yield, never both. Plenty of real tasks don't work that way: a headline can be about "earnings," "guidance," and "litigation" simultaneously. Multi-label classification handles the case where zero, one, or several labels can be true at once, which changes both how you model it and how you measure success.

The analogy: tagging an email versus filing it in one folder

Ordinary classification files a letter into exactly one folder. Multi-label classification tags an email with any combination that applies — "urgent," "finance," "needs-reply" can all be true at once, none excluding the others. A good multi-label method should exploit correlation between tags — "urgent" and "needs-reply" tend to co-occur — not just predict each in isolation.

The mechanics: from independent binary calls to label-aware ones

The simplest approach, binary relevance, trains one independent binary classifier per label:

$$P(\text{label}_k = 1 \mid x) = \sigma(w_k^\top x), \quad \text{predicted labels} = \{k : P(\text{label}_k=1\mid x) > 0.5\} .$$

In plain English: instead of one distribution across mutually exclusive classes summing to 1, you get $K$ independent yes/no probabilities, each high or low regardless of the others. Binary relevance ignores label correlation entirely; classifier chains instead feed each label's predicted probability as an extra input to the next label's classifier, letting later predictions condition on earlier ones.

Worked example: tagging a news headline

"Regulator opens probe into merger financing, shares fall" against three tags: earnings, M&A, litigation. Binary relevance scores independently: $P(\text{earnings})=0.05$ (not applied), $P(\text{M\&A})=0.71$ (applied), $P(\text{litigation})=0.62$ (applied) — giving {M&A, litigation}. A classifier chain, having learned litigation co-occurs strongly with M&A, might use the M&A prediction as input when scoring litigation, pushing a borderline 0.48 up to 0.62 — capturing that merger-related litigation is far more likely once a story is already known to be about M&A, something binary relevance can't represent.

Toggle flexibility above and think of each label as its own boundary drawn independently in binary relevance, versus a chain where each later boundary is allowed to bend based on what earlier labels already decided.

What this means in practice

Multi-label problems need multi-label-aware metrics: Hamming loss (fraction of individual label predictions wrong) and subset accuracy (fraction of examples where the entire predicted set exactly matches) tell different stories — low Hamming loss can coexist with rarely getting every label in a set exactly right. Whether label correlation is worth modeling explicitly depends on how strongly labels co-occur in your data.