Focal Loss for Imbalanced Targets

Imagine training a fraud detector where 1 in 1,000 transactions is fraudulent. A model that has already learned to confidently flag the other 999 as "not fraud" keeps generating loss (and gradient) from those easy examples on every single batch — even though the model already gets them right — simply because ordinary cross-entropy penalizes any gap between predicted probability and 1, however small. That accumulated gradient from thousands of easy, already-correct examples can drown out the comparatively rare, useful signal coming from the hard examples the model is still getting wrong. Focal loss fixes this by making the per-example loss shrink automatically once a prediction is already confident and correct.

The analogy: a teacher's attention in a classroom

A good tutor doesn't spend equal time re-explaining a problem to a student who already nailed it and one who is still stuck — attention flows to the students who need it. Ordinary cross-entropy is a tutor who insists on reviewing every student's answer with equal weight, even the ones who've mastered the material. Focal loss is the tutor who notices "the model already got this one right with 99% confidence" and automatically spends far less loss, and therefore far less gradient, re-teaching it, redirecting that budget toward the hard cases.

The formula

For binary classification, let $p_t$ be the model's predicted probability for the true class (so $p_t$ close to 1 means a confident, correct prediction). Ordinary cross-entropy is $-\log(p_t)$. Focal loss adds a modulating factor:

$$\text{FL}(p_t) = -(1-p_t)^{\gamma} \log(p_t)$$

In words: take the ordinary cross-entropy loss and multiply it by $(1-p_t)^\gamma$, a term that is close to 0 when $p_t$ is close to 1 (an easy, correct example) and close to 1 when $p_t$ is small (a hard or wrong example). The focusing parameter $\gamma$ (commonly 2) controls how aggressively easy examples get down-weighted — larger $\gamma$ suppresses them harder.

Worked example 1: an easy example versus a hard example, $\gamma = 2$

Easy example: the model predicts $p_t = 0.95$. Ordinary cross-entropy: $-\log(0.95) = 0.051$. Focal loss: $(1-0.95)^2 = 0.0025$, so $\text{FL} = 0.0025 \times 0.051 = 0.000128$ — about 400 times smaller than plain cross-entropy.

Hard example: the model predicts $p_t = 0.4$ (getting it mostly wrong). Ordinary cross-entropy: $-\log(0.4) = 0.916$. Focal loss: $(1-0.4)^2 = 0.36$, so $\text{FL} = 0.36 \times 0.916 = 0.330$ — reduced, but only by a factor of about 2.8, far less suppression than the easy example got.

Worked example 2: how the ratio shifts training's center of gravity

Under plain cross-entropy, the easy example contributes $0.051$ and the hard one $0.916$ — the hard example already dominates by about 18×. Under focal loss, the easy example contributes $0.000128$ and the hard one $0.330$ — a ratio of about 2,580×. Aggregated over a batch that is mostly easy examples (as in the 999-to-1 fraud case), that shift is decisive: the total gradient from thousands of easy, correct predictions collapses toward zero, while the gradient from the rare hard cases stays comparatively large.

Drag the exponent on the curve below (a power-law shape, matching $(1-p_t)^\gamma$) to see how sharply the down-weighting effect grows as $\gamma$ increases and $p_t$ moves away from 1.

What this means in practice

Focal loss was designed for dense object detection, where background regions vastly outnumber actual objects, but the same imbalance shows up constantly in finance: rare regime shifts, rare defaults, rare large-move labels swamped by the "nothing happened" majority class. It's a genuine alternative to naive class-weighting, because it adapts per-example rather than per-class — an easy minority-class example still gets suppressed, and a hard majority-class example still gets emphasized, which fixed class weights cannot do.