The seven ML ideas
everyone confuses.

Every one of these seven pairs answers a different question, and they stack. You can't sensibly pick a metric before you know whether you're classifying or regressing, and you can't diagnose an ensemble before you can tell bias from variance. Read them in order and each one hands you the vocabulary for the next.

Ask five people what "machine learning" means and you'll get five vague answers, but underneath almost all of it there are only three classic ways to actually teach a machine — and the split is the same one you'd use to teach a child. You can drill it with flashcards that have the answer on the back. You can hand it a pile of unsorted objects and let it work out the groupings itself. Or you can turn the whole thing into a game with a score and let it learn from winning and losing. Which one applies to your problem isn't a matter of taste — it's decided entirely by what you're able to give the model to learn from.

Supervised learning: labeled examples

Supervised learning is the flashcard method. You give the model thousands of examples where the correct answer is already attached — this photo is labeled cat, this email is labeled spam, this house sold for this exact price. The model's whole job is to learn the mapping from input to answer, so that when it sees a new, unlabeled input it can produce the right answer on its own.

Supervised learning is the workhorse of practical ML because it's direct: you know exactly what you're optimizing for, and you can measure how wrong the model is against labels you already hold. The catch is that someone has to write every flashcard. ImageNet — the dataset that kicked off the deep learning era — required roughly 14 million images to be hand-annotated through crowdsourced labor over several years. That cost is the single biggest practical constraint on supervised learning, and it's why so much of modern ML is an attempt to route around it.

Unsupervised learning: find the pattern yourself

Unsupervised learning removes the answer key entirely. You hand the model raw, unlabeled data and it has to find structure on its own — nobody tells it what the "right" grouping is, because there isn't one written down anywhere. It's the equivalent of dumping a toy box on the floor and letting a kid sort it by color or shape with no one naming the piles.

That self-discovered structure shows up in three recognizable forms. Clustering groups similar points without being told what the groups mean — k-means is the canonical algorithm, and you can watch it converge step by step in the k-means clustering lab. Dimensionality reduction squeezes data down to its essential structure, discarding redundancy (PCA, autoencoders, UMAP). Anomaly detection flags points that don't fit the pattern the rest of the data follows — useful for catching fraud or a failing sensor without ever having seen a labeled example of "fraud."

The advantage is that unsupervised learning runs on the enormous volume of raw data that already exists, with none of the labeling cost. The tradeoff is that you don't get a specific answer back — you get structure, and it's on you to decide what it means. This is also why unsupervised results are hard to evaluate: there's no ground truth to score against, so you fall back on proxy metrics (silhouette score, reconstruction error) that only loosely track whether the structure is useful.

Reinforcement learning: learn from reward and penalty

Reinforcement learning throws out both the flashcards and the toy box. There's no fixed dataset at all — just an agent, an environment, and a goal. The agent takes an action, the environment responds with a reward or penalty and a new state, and through repeated trial and error the agent learns a policy: a strategy for choosing actions that maximizes total reward over time.

This is how a dog learns tricks for treats, how AlphaGo taught itself to beat the best human players by playing itself millions of times, and how a simulated robot learns to walk without falling. None of it comes from labeled examples of "the correct move." It comes from feedback earned by acting. Two properties make RL genuinely different: the agent generates its own data by interacting, and the reward is often delayed — a chess move's real consequence lands forty moves later, which is the credit-assignment problem RL exists to solve.

Show a model a photo of a dog and you can ask it two very different kinds of questions. "What animal is this?" wants a label pulled from a short list. "How old is it?" wants a number that could land almost anywhere. That split — which bucket, or how much — is the first fork in the road for nearly every supervised problem, and it decides the model's output layer, its loss function, and how you'll score it, before you write a line of training code.

Classification: sort into a bucket

Classification sorts an input into one of a fixed set of discrete categories. Spam or not spam. Cat, dog, or bird. Fraud or legitimate. The output is always a label from a finite, predetermined list — there's no "spam-and-a-half," no answer that falls between two categories. Under the hood, a classifier is carving the input space into regions, one per bucket; the perceptron playground lets you drag points around and watch a linear boundary move in response, which is the whole idea in its simplest form.

Classifiers are trained against cross-entropy loss, which measures the negative log of the probability the model assigned to the correct answer. The shape of that penalty is the point — it grows without bound as the model becomes confidently wrong:

That asymmetry is deliberate. Cross-entropy doesn't just want the right answer, it wants calibrated confidence, and it punishes swagger harder than uncertainty. This is why a model trained with cross-entropy usually produces more trustworthy probabilities than one trained to maximize raw accuracy.

Regression: predict a number

Regression answers a different question: not which one, but how much. It predicts a continuous quantity — a point that can fall anywhere on a number line. Tomorrow's temperature, a house price, a person's age from a photo. There are no buckets to sort into at all, so a regression model fits a curve through the data rather than drawing boundaries between classes, and you score it by how far off it was rather than whether it was strictly right.

Predicting $402,000 for a house that sold for $400,000 is a much better outcome than predicting $250,000, even though neither is exactly correct. Squared error captures that directly — and the squaring is not cosmetic:

Squaring makes one catastrophic miss cost more than thousands of small ones — great for suppressing large errors, terrible when your data has genuine outliers. That's what MAE and Huber loss exist to fix.

Both losses are optimized the same way: compute the gradient of the loss with respect to every parameter and step downhill. The gradient descent lab shows that descent on a live surface, and the loss landscape lab shows why the shape of the surface — not just the loss value — determines whether the optimizer ever gets there.

The twist: classifiers output a number first

Here's where the line blurs. A classifier almost never jumps straight to a bare category. Internally it first produces a number — a probability like 0.92 — and only then applies a threshold to turn that probability into a bucket. Above the threshold, call it spam; below it, don't.

That's exactly why logistic regression, despite the word "regression" in its name, is a classification algorithm. It regresses onto a probability between 0 and 1, and a threshold classifies that probability into a label. The regression part is real; it's just the first half of the pipeline. It works in reverse too: any regression output can be turned into a classification by bucketing the number after the fact — turning a predicted price into "cheap," "mid-range," or "pricey." Holding onto that intermediate probability rather than throwing it away is what makes section 04 possible at all.

Train a model, watch its training accuracy climb, and it's tempting to assume you're winning. Often you're not: pushing training accuracy higher can make the model worse on the unseen data you actually care about. That paradox exists because there are two completely different ways a model can miss, and they pull in opposite directions.

Bias: consistently off-target

A high-bias model is too simple for the pattern it's trying to learn. It makes strong, often wrong, assumptions about the shape of the data — a straight line forced through a curved relationship — and no amount of extra training data will fix that, because the model simply can't represent the real function. This is underfitting, and its signature is that it shows up on both training and test data. The model isn't memorizing noise; it never learned the signal in the first place.

Variance: inconsistent across datasets

A high-variance model sits at the opposite extreme: flexible enough to memorize the training data's every quirk and every bit of random noise, not just the underlying signal. This is overfitting. It aces the data it trained on and falls apart the moment it sees anything new. Retrain it on a slightly different sample of the same population and it lands somewhere completely different — that instability is the variance.

The numbers

Here's what the two look like in a training log. Fit polynomials of increasing degree to the same noisy dataset, and watch training RMSE and test RMSE diverge:

Read the table the way a practitioner does. Both numbers high and close together means bias — add capacity. Train low, test high, wide gap means variance — add data or regularization. The degree-12 row is the whole lesson in one line: it fit its training set seventeen times better than the straight line and generalized worse than it.

Why you can't minimize both

Bias and variance move in opposite directions as you change model complexity. More expressive model → lower bias (it can represent more), higher variance (that same flexibility latches onto noise). Simpler model → the reverse. Expected test error decomposes cleanly:

Plot that sum against complexity and it traces a U. Too simple is high on the left, too complex is high on the right, and the best model sits at the bottom — where the sum is smallest, not where either term is zero.

That reframes the goal entirely. You were never trying to eliminate bias or eliminate variance; you're looking for the minimum of their sum. And note the σ² floor: if your labels themselves are noisy — two annotators disagreeing 8% of the time — no model can beat 92% no matter what you do. Knowing that number stops you from burning a quarter chasing an impossible target.

Regularization deserves a note because it's the most direct lever on variance: an L2 penalty shrinks weights, tree pruning caps depth, dropout randomly disables units during training. All of them buy a little bias to shed a lot of variance. Which lever fires first also depends on the optimizer — you can race several against each other on the same surface in the optimizer race lab, and see how activation choice reshapes what the network can represent in the activation functions lab.

A classifier that's "98% accurate" can still be useless, because accuracy hides the detail that matters: there are two completely different ways to be wrong. A spam filter can flag a real email you needed — a false alarm — or let actual spam through — a miss. Lumping those into one number throws away exactly the information you need.

A real confusion matrix

Take 10,000 emails, of which 400 are genuinely spam (4% — realistic class imbalance). Your filter flags 350 of them, and 300 of those flags are correct:

Precision looks down the flagged column; recall looks across the actually-spam row. Same 300, different denominator.

Now the punchline. A model that flags nothing at all — TP 0, FP 0, FN 400, TN 9,600 — scores 96% accuracy on this dataset while being worth precisely zero. Your real model beats it by only 2.5 accuracy points, which makes accuracy nearly useless as a signal here. That's the accuracy paradox, and it's why imbalanced problems are always reported with precision and recall instead.

The threshold is the dial

Precision and recall aren't independent knobs — they're two ends of the same dial: the decision threshold. Most classifiers output a suspicion score (section 02), and the threshold decides how much suspicion is enough to flag. Same model, same weights, three thresholds:

Nothing about the model changed across those three rows. Moving the threshold doesn't make a classifier better — it chooses which mistake you'd rather make more of. Note how brutally precision collapses at 0.20: catching the last 80 spam messages cost 550 extra false alarms, because there are 9,600 negatives and even a 6% false-positive rate on them swamps the 400 positives. Under heavy imbalance, precision is dominated by the size of the negative class, which is exactly why ROC-AUC can look reassuring while the precision-recall curve tells you the model is unusable.

Which one should you optimize?

There's no universally best threshold, because the right setting depends on the cost of each mistake, not on chasing the highest accuracy.

• A cancer screening test wants recall. Missing a real case is catastrophic, so you accept more false alarms — healthy patients who get a follow-up test — in exchange for almost never missing someone sick.
• A "delete forever" spam filter wants precision. Destroying one real email is unacceptable, so you'd rather let some spam through.
• Ranked results (search, recommendations) want precision@k: only the top handful is ever seen, so recall over the whole corpus is irrelevant.

When both errors genuinely matter and you need one number to compare models by, use F1, the harmonic mean. Because it's harmonic rather than arithmetic, it punishes lopsidedness hard: the 0.20-threshold row above averages to 66.9% arithmetically but scores just 0.551 on F1. If the two errors have genuinely different costs, use F_β — β > 1 weights recall, β < 1 weights precision. The same framework carries directly into LLM evaluation, where these are the metrics behind pass rates and guardrail scoring; the LLM evals interview handbook works through that version, and agent evals extends it to multi-step traces where a single "positive" spans a whole trajectory.

Grade a student on one random exam and the grade can lie — a lucky topic or an unlucky night's sleep says nothing reliable about what they know. Give them five exams on different material and average the results, and the grade starts to mean something. Measuring a model works the same way, and the choice between one exam and five is exactly the choice between a single hold-out set and k-fold cross-validation.

What's wrong with a single split

Carving off 20% as a test set, training on the rest, and scoring on the part the model never saw is a real improvement over scoring on training data. But a single split has two problems that don't go away just because the split was random.

Fragility. The score depends entirely on which rows landed in the test set. Shuffle differently and you get a different number, though nothing about the model changed. On a large dataset that noise washes out; on a small one it can swing enough to make a mediocre model look good.

Waste. Whatever rows you set aside never help the model learn. On a small dataset that's not a minor inefficiency — it's data held hostage to produce one score you don't fully trust.

How k-fold works

One idea fixes both: instead of one fixed split, rotate the test set through the whole dataset and average.

A single hold-out could have returned 0.79 or 0.88 from this exact model. Same model, a 9-point swing, decided by the shuffle.

That spread is the part people skip, and it's the most useful output. Five fold-scores clustered within a point of each other means you can trust the average. Five scores bouncing from 0.62 to 0.91 means the model's true performance is genuinely uncertain — something a single hold-out would have hidden behind one confident-looking figure. When you compare two models whose means differ by 0.01 and whose fold-standard-deviations are 0.03, the honest conclusion is that you can't tell them apart yet.

Choosing k, and the variants that matter

K-fold earns its keep in two situations. First, when data is scarce — small or expensive-to-label datasets are exactly where a single split's variance is largest and where you can least afford to spend rows on testing alone. Second, for hyperparameter tuning and model selection, where the score drives a decision rather than a report. Picking a learning rate or a champion model off one noisy hold-out score means your decision might just be chasing that split's luck. If you're tuning and reporting, you need nested CV: an inner loop that selects hyperparameters and an outer loop that scores the whole selection procedure — otherwise the reported number is optimistic for the same reason as tuning on the test set. Once the protocol is settled, freeze it and run it automatically on every change; wiring evaluation into the pipeline so a regression fails the build is what evals in CI covers.

A single deep decision tree memorizes its training data and overfits; a single shallow tree is too weak to learn much of anything. The fix in both cases isn't a cleverer tree — it's a crowd of them. But how you build that crowd matters, and there are two opposite philosophies. Bagging trains many models independently and averages their votes. Boosting trains models one after another, each aimed squarely at the last one's mistakes. They look like variations on one idea; they fix opposite failure modes — the exact two from section 03.

Bagging: independent models, averaged votes

Bagging — bootstrap aggregating — trains a batch of models fully independently. Each sees its own random bootstrap sample of the training data (drawn with replacement, so some rows repeat and roughly 37% are left out entirely), and often a random subset of features too. None of them know what the others are doing. Their predictions are then averaged for regression, or majority-voted for classification.

The intuition is a jury: each juror reasons from a slightly different slice of the evidence, so each is wrong sometimes — but their mistakes aren't correlated, so averaging cancels the noise. The math is exact and worth carrying around. Average n estimators that each have variance σ² and pairwise correlation ρ:

The second term vanishes as you add trees. The first term does not — it is a hard floor set by how correlated the members are.

Put numbers in it. With σ² = 1 and ρ = 0.1, one hundred trees give 0.10 + 0.009 = 0.109 — a 89% variance reduction. Go to a thousand trees and you get 0.1009: essentially nothing more, because you've hit the ρσ² = 0.1 floor. But drop correlation to ρ = 0.05 with the same hundred trees and variance falls to 0.0595 — nearly half again. That single equation explains the entire design of random forest: since adding trees saturates quickly, the real lever is decorrelating them, which is why random forest samples a random subset of features at every split rather than just bootstrapping rows. Bagging also parallelizes perfectly — no ordering dependency means every tree can train at once, on a different core or a different machine.

Boosting: sequential models, each fixing the last

Boosting inverts the approach. Models are trained one at a time, and each new model is deliberately pointed at whatever the previous ones got wrong. AdaBoost does this by reweighting examples — cases the ensemble currently misses get more weight, forcing the next model to attend to them. Gradient boosting, the modern default, fits each new model to the residual errors left over from the ensemble so far, which is literally gradient descent in function space. The models are combined as a weighted sum, each scaled by a learning rate (typically 0.01–0.1) so no single stage can overcorrect.

The intuition is a relay of specialists: the first model takes a rough pass, the second's entire job is to clean up what the first missed, the third cleans up what's left. Because each stage explicitly hunts the ensemble's current errors, boosting is exceptional at turning weak learners into a strong one — it attacks bias. This is why gradient-boosted trees (XGBoost, LightGBM, CatBoost) still win most tabular competitions. Two costs: it's inherently sequential (stage n needs stage n−1's residuals, so you can't parallelize across models, only within a tree), and because it keeps chasing whatever it currently gets wrong, it will eventually chase noise.

That "more rounds" row is the most practically important difference. Adding trees to a random forest is monotonically safe: variance keeps falling toward the correlation floor and then flattens, so "how many trees" is a compute decision, not a tuning decision. Adding rounds to a boosted model is not safe — validation error falls, bottoms out, then climbs as the ensemble starts fitting noise, so the number of rounds is a genuine hyperparameter that must be chosen by early stopping on a validation fold. When ensembles get large enough that a single machine can't hold them, or when the base learner is itself a neural network, the coordination problem becomes its own subject — see distributed training.

Every classifier learns from the same labeled data, but it can learn two fundamentally different things from it. Show a model a thousand labeled photos of cats and dogs, and it can either learn just enough to tell them apart, or it can learn what a cat actually looks like and what a dog actually looks like. The first models the boundary between classes. The second models the classes themselves. That choice determines not just how accurate the model is, but what it's even capable of.

Discriminative: learn the boundary

A discriminative model answers one narrow question directly: given this input, which class is it? Formally it learns P(y | x) and models that decision boundary as directly as possible, without representing anything else about the data.

The narrowness is a strength. Logistic regression, SVMs, and most neural-network classifiers are discriminative, and because they pour all their capacity into exactly where the classes differ, they're usually the sharper pure classifiers. A discriminative cat-vs-dog model never learns what fur texture looks like in general — only which side of the line an image falls on. It's a critic: excellent at telling two things apart, with no ability to describe or produce either.

Generative: learn the distribution

A generative model takes on a harder job: learning P(x | y), the distribution the inputs themselves come from, per class — what the space of "cat" images actually looks like, and separately what "dog" looks like. To classify, it doesn't look for a boundary at all; it asks which distribution the input more plausibly came from, via Bayes' rule:

The denominator doesn't depend on y, so it drops out. A generative classifier only needs the class-conditional likelihood and the prior.

A concrete pass: a Naive Bayes spam filter sees the word "invoice." Suppose P(invoice | spam) = 0.30, P(invoice | ham) = 0.02, and the prior is P(spam) = 0.04 from section 04's dataset. Then the spam score is 0.30 × 0.04 = 0.012 and the ham score is 0.02 × 0.96 = 0.019 — so despite "invoice" being 15× more likely in spam, a single occurrence still isn't enough to overcome a 24:1 prior against. Multiply in a second suspicious word at the same 15:1 ratio and spam wins decisively. That's the whole mechanic, and it's why priors matter so much on imbalanced data.

Naive Bayes and Gaussian mixture models are the classic generative classifiers. But the reason this distinction matters in 2026 is the other branch: because a generative model has modeled what each class actually looks like, it can also produce a new example of one. That's the core move behind GANs, diffusion models, and large language models — each learns the distribution of its training data, which is exactly what lets it generate an image or a sentence that never existed.

Why generative models can create and discriminative ones can't

The gap comes directly from what each bothers to learn. A discriminative model only represents the boundary, so there's nothing inside it to sample from — it has no notion of a "typical" member of a class, only which side of a line an input falls on. A generative model has already built a representation of the full distribution, so drawing a new sample is a well-defined operation: pick a class, then generate a plausible instance.

That distributional knowledge pays off twice more. It can flag an input as an outlier when it looks improbable under every distribution it knows — something a discriminative model has no basis to judge, since it never modeled what "normal" is. And because it extracts richer structure from each example, it often gets more out of a small labeled dataset. (An LLM is the extreme case: it models P(x) over text with no labels at all, and the classification ability falls out as a side effect.)

The trade-off is real. Modeling an entire distribution is a harder problem than modeling one boundary, and that extra effort doesn't always convert into classification accuracy — for a straightforward classification task, a discriminative model often wins precisely because it spends no capacity on parts of the problem the task doesn't need.

The practical rule: if the job is purely to tell classes apart, a discriminative model is usually the better tool — cheaper to train and typically sharper, because every bit of capacity goes toward the one decision that matters. If the job requires you to generate, detect the unseen, or work with few labels, you need a model that has learned what the data looks like, not just where its classes end. That is also the answer to why modern generative AI is called generative: a GAN, a diffusion model, and an LLM are all, by definition, modeling the distribution of their training data — and that modeling is precisely what lets them create something new rather than merely judge something given.