One exam,
or five?

from sklearn.model_selection import (
    KFold, StratifiedKFold, GroupKFold,
    TimeSeriesSplit, cross_val_score)
import numpy as np

# the one-liner most people want
s = cross_val_score(model, X, y, cv=5, scoring="f1")
print(f"{s.mean():.3f} ± {s.std():.3f}")   # 0.836 ± 0.033

# the same thing, unrolled
kf = KFold(n_splits=5, shuffle=True, random_state=0)
scores = []
for tr, va in kf.split(X):
    model.fit(X[tr], y[tr])              # k trainings!
    scores.append(score(model, X[va], y[va]))
print(np.mean(scores), np.std(scores))

# imbalanced labels → keep the class ratio per fold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
# correlated rows (same patient/user) → keep groups together
cv = GroupKFold(n_splits=5)     # .split(X, y, groups)
# temporal data → never train on the future
cv = TimeSeriesSplit(n_splits=5)

Picking k, and the four splits that aren't plain k-fold

These are the lab’s fixed row scores, so you can check every number against the widget. With k = 5 each fold is four consecutive rows. A fold’s score is the mean of its four row scores.

The mean is 0.836. The spread depends on which formula you use. Divide the squared deviations by k and you get the population std, 0.033, which is what the page reports and what np.std gives by default. Divide by k − 1 and you get the sample std, 0.037, which is the default in pandas. Say which one you used. Also look at the row scores inside each fold: F2 holds both 0.93 and 0.68. Fold scores are averages, so they move less than single rows do, and that is why the LOOCV row of the cost ladder shows the much wider ± 0.077.

• Preprocessing before the split

  Fit a scaler, an imputer, feature selection or a vocabulary on all the rows, then cross-validate, and every validation fold has already shaped the features. Put the preprocessing and the model in one scikit-learn Pipeline and pass that to cross_val_score, so each fold fits its own.

• Oversampling before the split

  Duplicate or synthesise minority rows (SMOTE, for example) and then split, and copies of the same row land on both sides. The model is then tested on near-duplicates of rows it trained on. Resample inside each training fold only, and never touch the validation fold.

• Leaving KFold unshuffled on sorted data

  scikit-learn’s KFold does not shuffle unless you pass shuffle=True. If the file is sorted by label, region or date, each fold becomes one slice of that ordering, and the scores measure the sort, not the model. Shuffle with a fixed random_state, unless the order is time, in which case use a time-series split.

• Calling fold noise a win

  Model B scores 0.842 and model A 0.836, with a fold std of 0.033. That gap is far smaller than the wobble. Score both models on the same folds, compare them fold by fold, and only believe a gap that holds on most folds.

• Shipping one of the fold models

  Cross-validation estimates how well your recipe works. Each fold model saw only 80% of the data. Once the recipe is chosen, refit it on all the rows and ship that. The CV score is the estimate for the refit model. GridSearchCV does this refit for you by default.

• Hyperparameter search

  GridSearchCV and RandomizedSearchCV run a full k-fold for every candidate. A grid of 20 settings with cv=5 is 100 training runs, plus one refit at the end. This is where the k× cost from the ladder above actually hurts, and why people shrink the grid before they shrink k.

• Stacking and out-of-fold predictions

  cross_val_predict returns, for every row, the prediction from the one model that did not train on it. Those out-of-fold predictions can train a second-level model without leakage. Training it on in-sample predictions instead teaches it to trust overfit base models.

• Small LLM eval sets

  Say you have 200 labelled examples and are choosing between prompts. That is the small-data case this page is about. Compare the candidates on the same folds or resamples and report the spread. If you also tuned the prompt on those examples, keep a held-back slice to report on.

• The interview version

  “You have 2,000 labelled rows. How do you evaluate?” A strong answer covers four points. Use stratified 5-fold, grouped by user if users repeat. Fit preprocessing inside the folds. Report the mean ± std. Use nested CV, or a final untouched test set, if the same data picked the hyperparameters.