Gradient Boosting
also: XGBoost · LightGBM
An ensemble that builds trees sequentially, each correcting the last one’s errors — the top performer on tabular data.
Gradient boosting builds an ensemble of trees sequentially, where each new tree is trained to correct the residual errors of the trees so far. It adds weak learners one at a time in the direction that most reduces the loss — libraries like XGBoost, LightGBM, and CatBoost made it the dominant approach for tabular data and Kaggle competitions.
Worked example: the first tree makes rough predictions; the second learns to predict the first’s errors; the third corrects the remaining error, and so on — the sum converges to a strong model. Gotcha: because it fits errors sequentially, gradient boosting can overfit if you add too many trees or too high a learning rate — unlike random forests, where more trees only help — so it needs early stopping and careful tuning of tree count, depth, and learning rate.