AdaBoost and Exponential Loss

A single shallow decision tree — sometimes just a one-split "stump" — is a weak classifier: barely better than a coin flip on its own. AdaBoost (Adaptive Boosting) chains many weak classifiers together, and the trick that makes the chain strong is what happens between rounds: every training example that the current classifier got wrong has its weight increased before the next classifier is trained, so the next one is forced to pay attention to exactly the cases the ensemble is still failing on.

The weight update and why it's exponential

After round $t$, each weak classifier gets a vote strength $\alpha_t = \frac{1}{2}\ln\left(\frac{1-\epsilon_t}{\epsilon_t}\right)$, where $\epsilon_t$ is its weighted error rate. In words: a classifier that gets almost everything right ($\epsilon_t$ near 0) earns a large positive vote weight; one that's barely better than random ($\epsilon_t$ near 0.5) earns a vote weight near zero.

Each training point's weight is then updated as $w_i \leftarrow w_i \times e^{\alpha_t}$ if it was misclassified, or $w_i \times e^{-\alpha_t}$ if it was classified correctly, then all weights are rescaled to sum to 1. In words: a wrong answer multiplies a point's importance up by a factor tied to how confident the round's classifier was; a right answer shrinks it. AdaBoost can be shown to be doing gradient descent on the exponential loss $L = e^{-y f(x)}$, where $y \in \{-1, +1\}$ is the true label and $f(x)$ is the ensemble's running weighted vote — a loss that grows explosively the more confidently wrong a prediction is.

Worked example

Round 1 trains a stump with weighted error $\epsilon_1 = 0.20$. Its vote weight is $\alpha_1 = 0.5 \ln\left(\frac{0.80}{0.20}\right) = 0.5 \ln(4) \approx 0.693$. A point it got wrong has its weight multiplied by $e^{0.693} \approx 2.0$ — twice as important in round 2. A point it got right is multiplied by $e^{-0.693} \approx 0.5$ — half as important. If round 2's stump then achieves a lower weighted error of $\epsilon_2 = 0.10$ (partly because those doubled-weight hard points are now easier to address directly), its vote weight climbs to $\alpha_2 = 0.5\ln(9) \approx 1.10$ — nearly 60% louder than round 1's vote, because it did better on the points that mattered most at that stage.

What this means in practice

AdaBoost's exponential loss is what makes it fast to converge on well-behaved data, but also what makes it fragile around mislabeled points: a single training row with a flipped label looks like the "hardest" point in the data and gets its weight blown up round after round, since the ensemble can never classify a wrong label correctly. In finance this shows up with noisy trade labels (a profitable trade tagged as a loss due to a settlement error) — AdaBoost will happily spend rounds chasing that one bad row.