One exam,
or five?

The problem with a single hold-out set

The naive way to check a model is to carve off a chunk of the data as a test set, train on everything else, and score on the part it never saw. That beats scoring on the training data — which only measures how well the model memorised its own answers — but a single split still has two problems that don't go away just because the split was "random".

The first is fragility. The score depends entirely on which rows landed in the test set. Shuffle differently and you get a meaningfully different number with nothing about the model changed. On a large dataset that noise mostly washes out; on a small one it swings enough to make a mediocre model look good, or a good model look mediocre, purely by chance.

The second is waste. Whatever rows you set aside never help the model learn. On a small dataset that isn't a minor inefficiency — it's data you can't afford to throw away, held hostage to produce one score you don't fully trust.

How k-fold fixes both

1. Split the dataset into k roughly equal folds.
2. Train on k − 1 folds, test on the one left out, record that fold's score.
3. Rotate: hold out a different fold, retrain from scratch, score again.
4. Repeat until every fold has been the test set exactly once — k rounds, k scores.
5. Report the mean, and the spread, of those k scores.

Across the full rotation every row is tested exactly once and trained on k − 1 times. The wasted-data problem simply disappears.

The whole thing in code

from sklearn.model_selection import (
    KFold, StratifiedKFold, GroupKFold,
    TimeSeriesSplit, cross_val_score)
import numpy as np

# the one-liner most people want
s = cross_val_score(model, X, y, cv=5, scoring="f1")
print(f"{s.mean():.3f} ± {s.std():.3f}")   # 0.836 ± 0.033

# the same thing, unrolled
kf = KFold(n_splits=5, shuffle=True, random_state=0)
scores = []
for tr, va in kf.split(X):
    model.fit(X[tr], y[tr])              # k trainings!
    scores.append(score(model, X[va], y[va]))
print(np.mean(scores), np.std(scores))

# imbalanced labels → keep the class ratio per fold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
# correlated rows (same patient/user) → keep groups together
cv = GroupKFold(n_splits=5)     # .split(X, y, groups)
# temporal data → never train on the future
cv = TimeSeriesSplit(n_splits=5)

The cost ladder

None of this is free. A hold-out trains one model; k-fold trains k. Every row of this table is computed from the same 20 fixed row-scores the lab above uses.