Cross-Validation: One Exam, or Five?
A single hold-out test set gives a fragile score that swings with which rows you happened to hold out, and wastes data the model never trains on. K-fold cross-validation rotates the test set across k folds and averages the scores for a far more stable estimate — while using every row for both training and testing.
A single hold-out split gives a fragile, high-variance score and wastes the held-out rows, since the model never trains on them. K-fold cross-validation splits the data into k folds, trains on k-1 of them and tests on the last, then rotates so every fold is the test set exactly once — averaging the k scores cancels out lucky or unlucky splits, and every row gets used for both training and testing. The cost is k times the compute, which is why it earns its keep most when data is scarce or for honest hyperparameter tuning.
Transcript
Grade a student on one random exam and the grade can lie — a lucky or unlucky day says nothing real. Measuring a model is the same. You need a fair score that uses all your data. That's cross-validation — so how many exams should you give?
The naive check: hold out a chunk of data as a test set, train on the rest, and score on the held-out part. But one split is fragile — the score swings with which rows you happened to hold out, and that data never helps the model learn.
Cross-validation fixes both. Split the data into k equal folds — say five. Train on four, test on the fifth. Then ROTATE: each fold takes its turn as the test set, exactly once. Five rounds, five scores — every point tested once, trained on four times.
Now average those five scores. Lucky and unlucky splits cancel out, so you get a far more stable estimate of true performance. And nothing is wasted — every single data point gets used for both training and testing. That's the whole win.
The catch: you train k models instead of one — so it costs k times the compute. Five folds, five trainings. On huge datasets, one hold-out is fine; the noise is tiny. Cross-validation earns its cost when data is precious.
So don't trust one lucky split. Rotate the test set across the folds, average the scores, and put every data point to work. It comes down to one question: grade the model on a single exam, or on five and take the average?
Keep going
Handbook51 LLM Evals Interview Questions →
Explore the topic
See this challenge alongside everything else on the same subject — handbooks, system designs, algorithms and tools, in one place.