Focal Loss

Standard classification training uses cross-entropy loss, which treats every example's error equally regardless of how confident the model already is. In a badly imbalanced problem — say, predicting a rare fraud flag or a rare tail event that occurs in 1% of rows — the model can rack up a low average loss just by confidently predicting "no event" every time, since the 99% of easy, correctly-classified examples swamp the loss contributed by the rare, hard positives.

Focal loss modifies cross-entropy by multiplying each example's loss by a factor $(1-p_t)^\gamma$, where $p_t$ is the model's predicted probability for the true class and $\gamma$ (commonly 2) is a tunable focusing parameter. When the model is already confident and correct, $p_t$ is close to 1, so $(1-p_t)^\gamma$ is close to zero and that example's contribution to the loss is squashed. When the model is wrong or unsure, $p_t$ is small, $(1-p_t)^\gamma$ stays close to 1, and the loss is barely reduced — training keeps pushing hard on exactly those examples.

Concretely, with $\gamma = 2$: an easy example correctly predicted with $p_t = 0.95$ gets its loss multiplied by $(1-0.95)^2 = 0.0025$, essentially removed from training, while a hard example with $p_t = 0.3$ gets multiplied by $(0.7)^2 = 0.49$, barely discounted — a roughly 200x difference in how much each example moves the gradient.