Bagging vs Boosting: A Jury, or a Relay?
Two opposite ways to combine many weak models into one strong ensemble: bagging trains models independently and averages their votes to cancel out variance, while boosting trains models sequentially, each one fixing the last one's mistakes, to reduce bias.
Bagging (random forest) trains many models independently on random slices of the data and averages their votes — like an independent jury whose individual mistakes cancel out, reducing variance. Boosting (gradient boosting) trains models sequentially, each one focused on what the last got wrong — like a relay of specialists, reducing bias but risking overfit to noise. They fix opposite failure modes, so which one helps depends on whether your problem suffers from variance or bias.
Transcript
One deep decision tree memorizes the training data and overfits. One shallow tree is too weak to learn much at all. The trick isn't a better single model — it's combining a whole crowd of them. But there are two opposite ways to do that.
That crowd is called an ensemble: many weak models pooled into one strong one. The magic is all in HOW you build and combine them. Do you train them in parallel, independent of each other — or in sequence, each one learning from the last? That single choice changes everything.
Bagging trains many models independently — each on its own random slice of the data — then averages their votes. It's a jury: each juror is wrong sometimes, but their independent mistakes cancel out. That tames overfitting — it reduces variance. Random forests work this way.
Boosting does the opposite. It trains models one after another, and each new one focuses on the examples the previous ones got WRONG. It's a relay of specialists, each cleaning up the last one's mistakes. That turns weak learners strong — it reduces bias. Gradient boosting works this way.
So they fix different problems. Bagging is parallel and robust — it calms a jittery, overfit model by averaging the noise away. Boosting is sequential and sharp — it squeezes out higher accuracy, but chase noise too hard and it can overfit. One cancels variance; the other attacks bias.
So don't just train one model — pool a crowd, and let the shape of that crowd match your problem. It comes down to one question: do you need to cut variance, or cut bias? A jury, or a relay?