Vibe Engines Learn
LABS  /  BIAS & VARIANCE

Fit the curve.
Watch it overfit.

Fifteen noisy points. One slider, from a straight line to a degree-12 polynomial. Training error falls the whole way — and somewhere in the middle, test error turns around and climbs. That turn is the entire bias–variance trade-off, and you can drive it yourself.

THE GIST · 20 SECONDS

Bias is how far off-target a model sits on average — too simple, it underfits and misses on train and test. Variance is how much it swings between training samples — too complex, it overfits, acing train and flopping on test. Total error = bias² + variance + irreducible noise, a U against complexity. Aim for the bottom of the U, not zero of either.

  • Biasoff-target on average
  • Varianceswings between samples
  • Underfitbad on train & test
  • Overfitgreat on train, bad on test

Drag degree to change model complexity, then hit resample a few times at degree 1 versus degree 12.

UNDER THE HOOD

What you just played, written down

Train a model and watch training accuracy climb, and it's tempting to assume you're winning. Often you're not — pushing training accuracy higher can make the model worse on the real, unseen data you actually care about. That paradox exists because there are two completely different ways a model can miss, and they pull in opposite directions.

Two ways to miss

Bias is how far off-target a model's predictions sit on average. A high-bias model is too simple for the pattern it's trying to learn: it makes strong, often wrong assumptions about the shape of the data — a straight line forced through a curved relationship — and no amount of extra training data will fix that, because the model isn't flexible enough to represent the real function. This is underfitting.

Variance is how inconsistent those predictions are across different training sets. A high-variance model is too complex, with enough flexibility to memorise the training data's every quirk and every bit of random noise, not just the underlying signal. It aces the data it was trained on and then falls apart the moment it sees anything new. This is overfitting.

THE DARTBOARD

Picture darts thrown at a board. Bias is how far the centre of the cluster sits from the bullseye — your aim. Variance is how spread out the darts are — your consistency. A high-bias model lands a tight little cluster in the wrong corner. A high-variance model scatters all over the board, averaging out about right and hitting nothing.

Why you can't minimise both

Bias and variance move in opposite directions as you change complexity. Make a model more expressive — more parameters, more depth, fewer restricting assumptions — and its bias drops, because it can now represent more complicated patterns. But its variance climbs, because that same flexibility lets it latch onto noise. Simplify it and the reverse happens.

THE DECOMPOSITION

E[(y − f̂(x))²]  =  Bias[f̂(x)]²  +  Var[f̂(x)]  +  σ²

  • Bias² — the systematic offset of the average prediction from the truth. Falls as complexity rises.
  • Variance — the expected spread of predictions around that average, across training sets. Rises as complexity rises.
  • σ² · irreducible noise — randomness in the data itself. No model removes it; it's the floor your test error can never beat. In this lab σ = 0.19, which is why the bottom of the U lands near 0.15–0.20 and never at zero.

Only the first two terms are yours to trade. Plot their sum against complexity and it traces a U: too simple sits high on the left (bias), too complex sits high on the right (variance), the best model sits at the bottom. The goal was never to zero out either one — it's to find the minimum of the sum.

Diagnosing it from the train/test gap

You almost never observe bias and variance directly. What you observe is two numbers — training error and held-out error — and the gap between them. That gap is the diagnostic:

High biasHigh variance
ModelToo simpleToo complex
Train errorHighVery low / zero
Test errorHigh (≈ train)High (≫ train)
GapSmallWide
SymptomUnderfits — bad on bothOverfits — great on train only
FixMore capacity / featuresMore data, regularise, ensemble

Both high, gap small → bias. The model misses on data it has already seen, so it never captured the signal. Train near zero, gap wide → variance. It captured the signal and the noise, and the noise doesn't transfer. If both are low and the gap is small, you're at the bottom of the U — ship it.

The fit, in code

import numpy as np

# least-squares polynomial fit — exactly what the slider does
c  = np.polyfit(x_train, y_train, deg)
p  = np.poly1d(c)

rmse = lambda a, b: np.sqrt(np.mean((a - b) ** 2))
tr = rmse(p(x_train), y_train)   # falls monotonically
te = rmse(p(x_test),  y_test)    # falls, then CLIMBS

# variance, measured: refit on many resamples of the
# same process and look at the spread of predictions
preds = [np.poly1d(np.polyfit(x, resample_y(), deg))(x_grid)
         for _ in range(50)]
var  = np.var(preds, axis=0)     # tiny at deg 1, huge at deg 12
bias = np.mean(preds, axis=0) - f_true(x_grid)
DEGREE 1σ ≈ 0.05barely moves on resample — high bias, low variance
DEGREE 12σ ≈ 1.4thrashes on resample — low bias, high variance
WHY 12 IS THE CLIFF

A degree-12 polynomial has 13 free coefficients against 15 points. There's almost exactly enough freedom to interpolate — thread the curve through every point, noise included — so training error collapses toward zero while the curve flails between the points and off the chart. That's variance you can see.

↑ Fixing high bias

Give it more capacity: a higher degree, a deeper tree, more layers or width, more features and interaction terms, a less restrictive functional form. Weaken the regularisation if it's over-constraining, and train longer if you stopped early. More data will not help — a line stays a line. Start at ML fundamentals.

↓ Fixing high variance

More data shrinks it directly — less room to latch onto any one sample's noise, and headroom to afford a richer, lower-bias model. Regularise: L2/L1 penalties, shallower trees, dropout, early stopping. Ensemble: bagging and random forests average independent errors away. Drop noisy features.

★ Where you'll meet it

Every model-selection decision is this curve: how many layers, how much dropout, when to early-stop, how big a LoRA rank. You only ever find the bottom of the U by measuring on held-out data — which is why an honest eval set is the whole game. See evals in CI.

QUICK CHECK

Did it stick?

FAQ

Bias and variance, answered

What is the bias–variance trade-off?

Bias is how far predictions sit from the truth on average; variance is how much they swing when you retrain on a different sample. More expressive models have lower bias and higher variance; simpler models the reverse. Because the two move in opposite directions as complexity changes, you can't zero both — you aim for the minimum of their sum.

How do I tell whether my model has high bias or high variance?

Compare training error with held-out error. Both high, small gap → high bias: it misses on data it has already seen, so it never learned the pattern. Train very low, test much higher → high variance: that wide generalisation gap is the signature of a model that memorised its training set.

What is the bias–variance decomposition?

Expected squared error on new data splits into three additive terms: bias² (systematic offset of the average prediction), variance (spread of predictions across training sets), and irreducible noise σ² (randomness in the data itself). Only the first two are yours to trade; σ² is the floor no model can beat.

How do I fix a high-variance model?

Shrink effective flexibility or add evidence: more training data, regularisation (L2/L1, shallower trees, dropout, early stopping), ensembling (bagging, random forests) so independent errors cancel, and dropping noisy or redundant features.

How do I fix a high-bias model?

Add capacity: higher degree or tree depth, more layers/width, better features, a less restrictive functional form, weaker regularisation, longer training. Note that more data does not fix bias — a straight line stays a straight line.

Why does test error rise while training error keeps falling?

Every extra degree of freedom lets the model bend closer to each training point, so training error falls monotonically and eventually hits zero by interpolation. Past a point those bends are fitting noise, which is different in fresh data — so the bends that helped on train actively hurt on test, and test error turns upward.

Doesn't deep learning break this? What about double descent?

The classical U still holds for a fixed model family with limited data — exactly what this lab shows. In heavily over-parameterised regimes, test error can fall again past the interpolation threshold, a phenomenon called double descent. It doesn't repeal the decomposition; it changes where implicit regularisation puts you on the complexity axis.

Finished this one? 0 / 61 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

ML Foundations

More Labs

Explore more from Vibe Engines

Handbooks →
Roadmaps →
System Designs →
AI System Designs →
Algorithms →
Paper Breakdowns →