Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

LABS  /  BIAS & VARIANCE

Fit the curve.
Watch it overfit.

Fifteen noisy points. One slider, from a straight line to a degree-12 polynomial. Training error falls the whole way — and somewhere in the middle, test error turns around and climbs. That turn is the entire bias–variance trade-off, and you can drive it yourself.

THE GIST · 20 SECONDS

Bias is how far off-target a model sits on average — too simple, it underfits and misses on train and test. Variance is how much it swings between training samples — too complex, it overfits, acing train and flopping on test. Total error = bias² + variance + irreducible noise, a U against complexity. Aim for the bottom of the U, not zero of either.

  • Biasoff-target on average
  • Varianceswings between samples
  • Underfitbad on train & test
  • Overfitgreat on train, bad on test

Drag degree to change model complexity, then hit resample a few times at degree 1 versus degree 12.

UNDER THE HOOD

What you just played, written down

Train a model and watch training accuracy climb, and it's tempting to assume you're winning. Often you're not — pushing training accuracy higher can make the model worse on the real, unseen data you actually care about. That paradox exists because there are two completely different ways a model can miss, and they pull in opposite directions.

Two ways to miss

Bias is how far off-target a model's predictions sit on average. A high-bias model is too simple for the pattern it's trying to learn: it makes strong, often wrong assumptions about the shape of the data — a straight line forced through a curved relationship — and no amount of extra training data will fix that, because the model isn't flexible enough to represent the real function. This is underfitting.

Variance is how inconsistent those predictions are across different training sets. A high-variance model is too complex, with enough flexibility to memorise the training data's every quirk and every bit of random noise, not just the underlying signal. It aces the data it was trained on and then falls apart the moment it sees anything new. This is overfitting.

THE DARTBOARD

Picture darts thrown at a board. Bias is how far the centre of the cluster sits from the bullseye — your aim. Variance is how spread out the darts are — your consistency. A high-bias model lands a tight little cluster in the wrong corner. A high-variance model scatters all over the board, averaging out about right and hitting nothing.

Why you can't minimise both

Bias and variance move in opposite directions as you change complexity. Make a model more expressive — more parameters, more depth, fewer restricting assumptions — and its bias drops, because it can now represent more complicated patterns. But its variance climbs, because that same flexibility lets it latch onto noise. Simplify it and the reverse happens.

THE DECOMPOSITION

E[(y − f̂(x))²]  =  Bias[f̂(x)]²  +  Var[f̂(x)]  +  σ²

  • Bias² — the systematic offset of the average prediction from the truth. Falls as complexity rises.
  • Variance — the expected spread of predictions around that average, across training sets. Rises as complexity rises.
  • σ² · irreducible noise — randomness in the data itself. No model removes it; it's the floor your test error can never beat. In this lab σ = 0.19, which is why the bottom of the U lands near 0.15–0.20 and never at zero.

Only the first two terms are yours to trade. Plot their sum against complexity and it traces a U: too simple sits high on the left (bias), too complex sits high on the right (variance), the best model sits at the bottom. The goal was never to zero out either one — it's to find the minimum of the sum.

Diagnosing it from the train/test gap

You almost never observe bias and variance directly. What you observe is two numbers — training error and held-out error — and the gap between them. That gap is the diagnostic:

High biasHigh variance
ModelToo simpleToo complex
Train errorHighVery low / zero
Test errorHigh (≈ train)High (≫ train)
GapSmallWide
SymptomUnderfits — bad on bothOverfits — great on train only
FixMore capacity / featuresMore data, regularise, ensemble

Both high, gap small → bias. The model misses on data it has already seen, so it never captured the signal. Train near zero, gap wide → variance. It captured the signal and the noise, and the noise doesn't transfer. If both are low and the gap is small, you're at the bottom of the U — ship it.

The fit, in code

import numpy as np

# least-squares polynomial fit — exactly what the slider does
c  = np.polyfit(x_train, y_train, deg)
p  = np.poly1d(c)

rmse = lambda a, b: np.sqrt(np.mean((a - b) ** 2))
tr = rmse(p(x_train), y_train)   # falls monotonically
te = rmse(p(x_test),  y_test)    # falls, then CLIMBS

# variance, measured: refit on many resamples of the
# same process and look at the spread of predictions
preds = [np.poly1d(np.polyfit(x, resample_y(), deg))(x_grid)
         for _ in range(50)]
var  = np.var(preds, axis=0)     # tiny at deg 1, huge at deg 12
bias = np.mean(preds, axis=0) - f_true(x_grid)
DEGREE 1σ ≈ 0.07barely moves on resample — high bias, low variance
DEGREE 12σ ≈ 0.68thrashes on resample — low bias, high variance

σ here is the spread of the fitted curve: sqrt(mean(var)) from the code above, over 500 resamples and x from −1 to 1. At degree 12 it is not even: about 0.17 in the middle of the range and up to 2.5 in the gaps between the outermost points, where the curve flails.

WHY 12 IS THE CLIFF

A degree-12 polynomial has 13 free coefficients against 15 points. There's almost exactly enough freedom to interpolate — thread the curve through every point, noise included — so training error collapses toward zero while the curve flails between the points and off the chart. That's variance you can see.

↑ Fixing high bias

Give it more capacity: a higher degree, a deeper tree, more layers or width, more features and interaction terms, a less restrictive functional form. Weaken the regularisation if it's over-constraining, and train longer if you stopped early. More data will not help — a line stays a line. Start at ML fundamentals.

↓ Fixing high variance

More data shrinks it directly — less room to latch onto any one sample's noise, and headroom to afford a richer, lower-bias model. Regularise: L2/L1 penalties, shallower trees, dropout, early stopping. Ensemble: bagging and random forests average independent errors away. Drop noisy features.

★ Where you'll meet it

Every model-selection decision is this curve: how many layers, how much dropout, when to early-stop, how big a LoRA rank. You only ever find the bottom of the U by measuring on held-out data — which is why an honest eval set is the whole game. See evals in CI.

THE PICTURE

The picture: the lab’s two error curves

best: degree 3, test 0.153 underfit: bias overfit: variance train test 0.5 0.25 0 1 3 6 9 12 polynomial degree · RMSE on the 15 train and 24 test points
This is the error chart the slider draws, for the starting sample. Train error only ever goes down, because each extra degree can only fit the 15 points as well or better. Test error bottoms out at degree 3 and then rises. Underneath, the decomposition does exactly what the theory says. Over 500 resamples, bias² is 0.111 at degree 1, 0.0025 at degree 3 and about 0 from degree 5 up. Variance grows the whole way: 0.005, 0.009, then 0.024 at degree 8 and 0.457 at degree 12.
BY THE NUMBERS

Worked example: reading the gap, degree by degree

These are the numbers the meters show for the starting sample, before any resample. The gap is test RMSE minus train RMSE.

DegreeTrain RMSETest RMSEGapReading
10.4060.357−0.049Both high, no gap: high bias. A line cannot bend into a sine.
20.4010.366−0.035No better. The truth is an odd function, so an x² term adds nothing useful.
30.1550.153−0.002Both low, no gap: the bottom of the U. Below the noise σ of 0.19 only by luck of this sample.
40.1330.164+0.031Train still falling, test already rising.
60.0990.205+0.106The gap opens. The extra bends fit noise.
90.0810.232+0.151Wider still, with 10 coefficients for 15 points.
110.0530.320+0.267Train has dropped well below the noise level. That is memorising.
120.0420.536+0.49413 coefficients for 15 points. The curve flails between them.

Two things stand out. First, the gap at degrees 1 to 3 is slightly negative. On 15 and 24 noisy points, test can come out a little better than train by chance, so read a small gap as “no gap”. Second, degree 2 buys nothing because 0.78 sin(3x) is odd, and only odd powers help. Over 500 resamples, bias² is 0.111 at both degree 1 and degree 2, and 0.0025 at both 3 and 4. Train error below σ = 0.19 is a warning in itself. No honest model can beat the noise on data it did not see, so a train number well under it means the noise was learned.

TRADE-OFFS

Trade-offs: what each fix costs

For this topic the columns mean: Time is the extra compute, Space is what else you pay or give up. A least-squares fit by QR, which the lab uses, costs about n·m² for n points and m = degree + 1 coefficients.

CaseTimeSpaceWhy
Lower the degree (or depth, or width)Cheaper: fewer coefficients to fitBias rises if you go past the bottom of the UAt degree 1 or 2 here, bias² is 0.111. That is about 44 times the 0.0025 at degree 3.
More training dataFit cost grows linearly in nCollecting and labelling it is usually the real costCuts variance only. It does nothing for degree 1, which stays a line.
L2 / L1 regularisationSame as the plain fit, plus a search over λλ is a new hyperparameter to tuneKeeps a flexible model but shrinks its coefficients, which trades a little bias for a lot less variance.
Bagging / random forestB models means B× training and B× predictionB× memory for the modelsAveraging errors that are independent cuts variance. Bias stays where it was.
k-fold cross-validation to pick the degreek fits per candidate: 12 degrees × 5 folds = 60 fitsNone at serving timeYou pay this once to choose. It replaces a single split that can pick the wrong degree.

Every fix for variance makes the model less able to chase any single sample. The only real choice is where that limit comes from: fewer parameters, smaller weights, more data, or averaging over many models.

WATCH FOR

Common mistakes

  • Choosing the degree on the test set

    Pick the degree with the lowest test error and that test number is no longer an honest estimate, because you chose it for being low. Choose on a validation set or by cross-validation. Touch the test set once, at the end.

  • Trusting one split on a small dataset

    Resample the lab’s noise 500 times and degree 3 wins on 405 of them (81%). Degrees 4 to 10 win the other 95. With little data, the “best” setting is itself noisy. Average over k folds before you believe it.

  • Collecting more data to fix bias

    If train error is as bad as test error, the model cannot represent the pattern. A thousand more points will give you a very confident straight line. Add capacity or better features first, then check the gap again.

  • Confusing variance with noise

    Noise (σ²) is in the data, and nothing removes it. Variance is a property of the model: how much its predictions move when the training sample changes. You can only see variance by refitting on different samples, which is what the resample button does.

  • Calling a near-zero train error success

    With labels as noisy as σ = 0.19, a train RMSE of 0.042 is not skill. It means the model has learned the noise. Compare train error with the noise level you expect, not with zero.

IN THE WILD

Where this shows up in production and interviews

  • Hyperparameter search

    Tree depth, regularisation strength, learning rate and number of layers are all positions on this curve. Tools like scikit-learn’s GridSearchCV do exactly what the table above prices: k-fold cross-validation over each candidate, keeping the one with the best held-out score.

  • Forests and boosting

    A random forest averages many deep, high-variance trees, so the average has much less variance. Gradient boosting goes the other way. It adds shallow, high-bias trees one at a time, each fixing what the last ones missed, so it cuts bias and needs early stopping to keep variance in check.

  • Early stopping in deep learning

    Training epochs act like the degree slider. Validation loss falls, flattens, then turns up while training loss keeps falling. Saving the checkpoint at the validation minimum is choosing the bottom of the U along the time axis.

  • The interview version

    “99% train accuracy, 70% validation. What do you do?” Name it as high variance from the gap. List the fixes in order of cost: regularise, simplify, add or augment data, ensemble. Say you would choose with cross-validation and keep a final test set untouched. Then flip it: “and if both were 70%?”

QUICK CHECK

Did it stick?

See every answer, with the reasoning

  1. Your model scores 0.02 RMSE on training data and 0.48 on held-out data. What do you have?

    Answer: High variance — the model is overfitting — Near-zero training error with a much larger test error is the signature of a wide generalisation gap: the model memorised the training set, noise included, and that noise does not repeat in fresh data. Reach for more data, regularisation, or an ensemble.

  2. A model scores 0.41 RMSE on train and 0.42 on test. What should you do?

    Answer: Give the model more capacity — more features, higher degree, more depth — Both errors are high and nearly equal, so the gap is small: this is high bias / underfitting. The model is missing on data it has already seen, meaning it cannot represent the pattern at all. More data will not help a straight line become a curve — more capacity will.

  3. Total expected error on new data decomposes into which three terms?

    Answer: Bias² + variance + irreducible noise — Expected squared error = Bias[f̂]² + Var[f̂] + σ². Only the first two respond to model choices; σ² is randomness in the data-generating process and sets the floor that even a perfect model cannot beat.

  4. Why does training error keep falling while test error turns and climbs?

    Answer: Extra flexibility starts fitting noise, which is specific to the training sample and does not transfer — Each extra degree of freedom lets the curve bend closer to every training point — eventually interpolating them exactly, so train error goes to zero. Past the sweet spot those bends are chasing noise, which differs in fresh data, so they actively hurt test error.

FAQ

Bias and variance, answered

What is the bias–variance trade-off?

Bias is how far predictions sit from the truth on average; variance is how much they swing when you retrain on a different sample. More expressive models have lower bias and higher variance; simpler models the reverse. Because the two move in opposite directions as complexity changes, you can't zero both — you aim for the minimum of their sum.

How do I tell whether my model has high bias or high variance?

Compare training error with held-out error. Both high, small gap → high bias: it misses on data it has already seen, so it never learned the pattern. Train very low, test much higher → high variance: that wide generalisation gap is the signature of a model that memorised its training set.

What is the bias–variance decomposition?

Expected squared error on new data splits into three additive terms: bias² (systematic offset of the average prediction), variance (spread of predictions across training sets), and irreducible noise σ² (randomness in the data itself). Only the first two are yours to trade; σ² is the floor no model can beat.

How do I fix a high-variance model?

Shrink effective flexibility or add evidence: more training data, regularisation (L2/L1, shallower trees, dropout, early stopping), ensembling (bagging, random forests) so independent errors cancel, and dropping noisy or redundant features.

How do I fix a high-bias model?

Add capacity: higher degree or tree depth, more layers/width, better features, a less restrictive functional form, weaker regularisation, longer training. Note that more data does not fix bias — a straight line stays a straight line.

Why does test error rise while training error keeps falling?

Every extra degree of freedom lets the model bend closer to each training point, so training error falls monotonically and eventually hits zero by interpolation. Past a point those bends are fitting noise, which is different in fresh data — so the bends that helped on train actively hurt on test, and test error turns upward.

Doesn't deep learning break this? What about double descent?

The classical U still holds for a fixed model family with limited data — exactly what this lab shows. In heavily over-parameterised regimes, test error can fall again past the interpolation threshold, a phenomenon called double descent. It doesn't repeal the decomposition; it changes where implicit regularisation puts you on the complexity axis.

Finished this one? 0 / 61 Labs done
NextCross-Validation: One Exam, or Five? Next on the “Judging a model” ladder · rung 3 of 3

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

ML Foundations

More Labs