LogitBoost and Real AdaBoost

Classic AdaBoost combines weak learners that each output a hard +1/-1 label, weighted by how accurate they were. Real AdaBoost and LogitBoost, introduced in the same influential 2000 paper, showed that boosting can instead be understood as fitting an additive logistic regression model, where weak learners output real-valued class probability estimates rather than hard labels, and the whole ensemble is trained by minimizing log-loss (the same loss logistic regression itself uses) instead of AdaBoost's exponential loss.

Worked example. A weak learner in classic AdaBoost might say "this loan applicant defaults" (a hard vote). The same weak learner under Real AdaBoost instead reports "this applicant has a 73% chance of default," and the ensemble combines these probability estimates using the log-odds scale, which behaves more smoothly with noisy or overlapping classes. LogitBoost takes this further: at each round it fits a weighted least-squares regression to "working responses" derived from the current model's residual log-odds error, exactly mirroring how ordinary logistic regression is fit by iteratively reweighted least squares — just done in stages, one weak learner at a time, instead of all at once.

This connection matters practically because exponential loss (classic AdaBoost) is quite sensitive to mislabeled training examples — an extreme outlier gets an exponentially growing weight — while log-loss-based variants like LogitBoost are more robust to label noise, at some cost in convergence speed.