Ensembles & Boosting

Read these first

1. 01Blending vs Stacking

   Both combine several models into one prediction by training a "meta" model on their outputs, but they split the data differently, and that difference changes how much you can trust the result.

2. 02Forward Stagewise Additive Modelling

   Build a strong model one small, greedy piece at a time, fit a weak model, see what it still gets wrong, fit another weak model to just that leftover error, and repeat, which is the shared skeleton underneath both boosting and simple additive forecasting.

3. 03Reliability Diagrams and Calibration Curves

   A reliability diagram checks whether a model's stated probabilities mean what they claim to mean, whether events it calls "70% likely" really happen about 70% of the time.

4. 04Why Ensembles Work: The Ambiguity Decomposition

   Averaging several models is not a vague good idea, for squared error there is an exact formula proving the ensemble is always at least as good as the average member, and the gap is exactly how much the members disagree.

5. 05Gradient Boosting as Functional Gradient Descent

   Gradient boosting is ordinary gradient descent, except the thing being updated at each step is not a vector of weights but an entire function, added to the model one small correction at a time.

Then the rest

1. 06A Practical Tuning Order for Gradient Boosting
2. 07Bias in Random Forest Feature Importance
3. 08Boosting and Label Noise
4. 09Ensemble Diversity and Model Correlation
5. 10Ensemble Variance as an Uncertainty Estimate
6. 11Explainable Boosting Machines
7. 12Histogram-Based Split Finding
8. 13LambdaMART and Ranking Objectives
9. 14Optimal Ensemble Weights
10. 15Out-of-Fold Predictions for Stacking
11. 16Platt Scaling
12. 17Random Forests vs Gradient Boosting
13. 18Regularization Knobs in Gradient Boosted Trees
14. 19Shrinkage and the Learning-Rate vs Tree-Count Tradeoff
15. 20Stochastic Gradient Boosting
16. 21Tree Depth and Interaction Order in Boosting
17. 22Tuning Random Forests: Trees, Depth and mtry
18. 23Which Learners Benefit From Bagging
19. 24Why Boosting Often Resists Overfitting
20. 25Custom Objectives and Gradients in Boosting
21. 26Newton Boosting and Second-Order Approximations
22. 27Ordered Boosting and Target Statistics
23. 28Quantile Regression Forests