Bias-Variance Tradeoff
The core tradeoff in ML: too simple a model underfits (high bias), too complex overfits (high variance).
The bias-variance tradeoff decomposes a model’s error into two sources. Bias is error from wrong assumptions — too simple a model that underfits and misses the real pattern. Variance is error from over-sensitivity to the training data — too complex a model that overfits its noise. Total error is minimized by balancing the two.
Worked example: fitting a straight line to a curve is high bias (underfits); fitting a wiggly degree-20 polynomial through every point is high variance (overfits); a moderate model that captures the shape without the noise sits at the sweet spot. Gotcha: more data reduces variance (letting you afford a more complex model), while regularization trades a little bias for a big drop in variance — and the classic tell of high variance is a large gap between training and validation error, of high bias, both being poor.