Bias vs Variance: The Two Ways a Model Misses (and Why You Can't Kill Both)
Bias is how far off-target a model sits on average — a too-simple model that underfits. Variance is how inconsistent it is across different training sets — a too-complex model that overfits. You cannot minimize both at once, so the goal is the minimum of their sum.
Bias is how far a model's predictions sit from the truth on average — a too-simple model underfits, missing the target on both train and test data. Variance is how much the model's predictions swing across different training sets — a too-complex model overfits, scattering wildly and flopping on fresh data. Test error decomposes into bias squared plus variance plus irreducible noise, a U-shaped curve against model complexity, so the goal is the minimum of the sum, not zero of either: more data or regularization shrinks variance, a more expressive model cuts bias.
Transcript
Here's a paradox that trips up every new ML practitioner: making your model MORE accurate on the training data often makes it WORSE on real, unseen data. Why? It's the bias-variance tradeoff. Picture throwing darts at a board — two totally different ways to miss. Understanding them is most of machine learning.
High BIAS is a model that's too simple. It makes strong, wrong assumptions and can't capture the real pattern in the data — so it underfits. On the dartboard, bias is how far your darts' CENTER sits from the bullseye — your aim. High bias means the whole cluster is off-target: bad on training AND new data.
High VARIANCE is the opposite: a model so complex it memorizes the training data — every quirk, every bit of noise. It overfits. Variance is how SPREAD OUT the darts are — your consistency. High variance means they scatter: retrain on slightly different data and the model lands somewhere completely different.
Here's the catch: you can't just minimize both. Make the model more complex and bias drops — but variance climbs. Simplify it, and the reverse happens. Your test error is bias squared plus variance, plus irreducible noise you can never remove — and it's U-shaped. The goal isn't zero of either: it's the minimum of the sum.
So the whole game is finding that sweet spot. More training data shrinks variance, which lets you afford a richer, lower-bias model without overfitting. Regularization reins in variance; a more expressive model can cut bias. You're always sliding along that curve, trading one kind of error for the other.
So there are two ways to miss: aiming wrong is bias, a shaky hand is variance. A great model needs both fixed — good aim AND a steady hand. Next time your model aces training but flops in production, you'll know: that's variance. Don't chase a perfect fit — aim for where your darts cluster tightest on the bullseye.