Blend the
examples.

Rung 1 regularises by perturbing the model — units vanish, so no unit can rely on another. mixup leaves the model untouched and perturbs the training set, and it is the only rung on this ladder that also perturbs the labels. Two examples are drawn, and the network is trained on a convex combination of both: a blend of the two images, paired with the same blend of the two one-hot labels.

The blending weight λ is drawn from a Beta distribution (in the authors’ reference code, once per batch rather than per example) whose single parameter α controls how aggressive the mixing is — small α keeps λ near 0 or 1, so most examples are nearly unmixed, and larger α pushes mass toward genuine half-and-half blends. Reported ImageNet recipes sit at the gentle end, in the region of α ≈ 0.2 to 0.4.

Stated in rung 1's units: mixup adds one hyperparameter (α) and no parameters, exactly like dropout, and it costs essentially nothing per step — one extra pair of tensor blends. What it buys is different in kind. Dropout tells the model not to over-rely on any one feature. mixup tells it what to output in between two training points: that the answer should move linearly as the input moves from one example to the other. That is a statement about the whole space between the data, not about the data.

Standard training minimizes loss at the exact training examples — and nowhere else. That's called empirical risk minimization, and it leaves the model free to do whatever it likes between the data points, which is most of input space. Neural networks tend to fill that void with jagged, wildly confident decision boundaries: they'll assign 99.9% confidence to regions they've never really seen, memorize noisy labels, and flip their answer under tiny adversarial perturbations. The trouble is precisely the space in between.

mixup (Zhang, Cisse, Dauphin & Lopez-Paz, 2018) addresses this with a startlingly simple idea: train on convex combinations of pairs of examples. Take two training points, blend their inputs with some weight λ, blend their labels with the same weight, and train on the result. This is vicinal risk minimization — you're minimizing loss not just at the data points but in the interpolated neighborhood between them, teaching the model to transition linearly from one class to another rather than snapping discontinuously. Two lines of code, no new parameters, and a measurably better model.

The crucial detail is that mixup blends the labels too, not just the inputs. If you averaged two images but kept a single hard label, you'd be lying to the model about a half-and-half picture. Instead the label becomes a soft target: a 70/30 image gets the target "0.7 of class A, 0.3 of class B." The blend is honest, and the loss (cross-entropy against the soft target) is meaningful.

At λ = 1 you recover example i exactly; at λ = 0, example j; in between, a genuine mixture with a matching soft label. Because the same λ weights both input and label, the mapping stays consistent — the model is told "this input is this fraction of the way from A to B, and so is its answer." Repeated over many random pairs, it learns to interpolate smoothly across the whole data manifold.

How aggressively should you mix? That's set by drawing λ from a Beta(α, α) distribution — a symmetric distribution on [0, 1] with a single hyperparameter α that shapes it:

Because Beta(α, α) is symmetric, its mean is always 0.5, but α decides whether the mass piles up at the extremes (gentle) or spreads toward the middle (aggressive). Small α keeps most blends close to a real example with a light dusting of a second; that's usually where the gains live. Push α too high and you drown every example in a 50/50 mush, which can underfit. It's a single dial trading off "stay near real data" against "explore the space between," and modest values tend to win.

The entire method is two equations sharing one mixing weight:

(x_i, y_i) and (x_j, y_j) are two examples with one-hot labels y. x̃ is the blended input; ỹ is the blended label (a valid probability distribution, since a convex combination of one-hot vectors sums to 1). Train on (x̃, ỹ) with the usual cross-entropy loss.

Two properties make it well-behaved. First, it's a genuine convex combination: the weights λ and 1−λ are non-negative and sum to 1, so x̃ lies on the segment between the two inputs and ỹ stays a proper probability distribution. Second, it degrades gracefully — at the endpoints it is ordinary training on a single example, so mixup only ever adds interpolated examples to the mix. The runnable version below implements the input blend, the label blend, and checks these endpoint and simplex properties.

Zhang, Cisse, Dauphin and Lopez-Paz present mixup as vicinal risk minimization with one specific vicinity, and their reference implementation is a few lines added to an ordinary training loop:

1. Get two batches, or one batch and a shuffled copy. The paper's code zips two data loaders. The authors' own implementation instead takes one mini-batch and mixes it with a randomly shuffled copy of itself, which they report works equally well and halves the input loading.
2. Draw one λ per batch. In the paper's code, λ ~ Beta(α, α) is sampled once per iteration and used for every pair in the batch. The vicinal distribution behind it is defined per example, so drawing λ per example is the same method in expectation.
3. Mix the inputs and the one-hot labels with that λ. x̃ = λx_i + (1 − λ)x_j and ỹ = λy_i + (1 − λ)y_j, with partners paired at random across all classes.
4. Train with the ordinary loss on the soft target. Cross-entropy against ỹ equals λ times the loss on y_i plus (1 − λ) times the loss on y_j, so many implementations keep integer labels and mix two losses.
5. Test on clean inputs. Nothing changes at inference; mixup only reshapes the training distribution.

What the vicinity replaces

Vicinal risk minimization, introduced by Chapelle and colleagues in 2000, trains on a neighbourhood around each example instead of on the point itself. The classic choice is a Gaussian vicinity: add small noise to the input and keep its label, which is what ordinary noise augmentation amounts to. mixup's vicinity is built from the data instead. The neighbours of an example are blends with other training examples, and the label moves with the input, which is why it can say something about the space between classes, where a Gaussian vicinity centred on one example and one label cannot.

Variants the authors tried and dropped

Mixing three or more examples with weights drawn from a Dirichlet distribution gave no further gain and cost more computation. Interpolating only between examples with the same label did not produce mixup's gains, so the effect depends on blending across classes. Pairing each example with a nearest neighbour instead of a random partner was weaker, and mixing hidden representations instead of inputs regularized less the deeper the layer.

The training setups

ImageNet-2012: ResNet-50, ResNet-101 and ResNeXt-101, trained with the large-batch Caffe2 recipe of Goyal and colleagues, a batch of 1,024 and a learning rate warmed from 0.1 to 0.4 over five epochs and then stepped down, for 90 or 200 epochs, with α = 0.2 for the ResNets and 0.4 for ResNeXt. CIFAR-10 and CIFAR-100: PreAct ResNet-18, WideResNet-28-10 and DenseNet-BC-190 for 200 epochs at batch 128, learning rate 0.1 divided by 10 at epochs 100 and 150 (60, 120 and 180 for WideResNet), weight decay 10^⁻⁴ and α = 1. Speech: the Google Commands set of 65,000 utterances in 30 classes, mixed as spectrograms, on LeNet and VGG-11. Tabular: six UCI data sets with a network of two 128-unit hidden layers. The paper also trains GAN discriminators on blends of real and generated samples, with λ as the target.

The paper casts mixup as vicinal risk minimization: instead of the empirical distribution of the n training points, the model is trained on a distribution of blends around each one.

• μ — the vicinal distribution: around training point i, every blend it can form with every point j, weighted by the Beta draw.
• Rν(f) — the empirical vicinal risk, the average loss over m samples drawn from μ; it replaces the average loss over the n raw points.
• λ·CE(…, yi) + (1 − λ)·CE(…, yj) — holds because cross-entropy is linear in the target, so a soft label and a mix of two hard-label losses are the same computation.

At the extremes, as α → 0 the Beta draws pile up at 0 and 1, μ collapses onto the training points and R_ν becomes ordinary empirical risk. In between, the loss on a blend cannot reach zero: the best prediction is ỹ itself and the remaining loss is its entropy, about 0.61 nats for a 70/30 blend, so training keeps pulling predictions between classes toward the blend's proportions rather than toward certainty.

The ImageNet gains grow with capacity and training length: 0.2 points for ResNet-50 at 90 epochs against 0.5 to 0.6 for the larger models, and at 200 epochs mixup cut ResNet-50's error a further 1.2 points while ERM stayed flat. α from 0.1 to 0.4 helped there; larger values underfit. With half the CIFAR-10 labels randomized, ERM memorized the noise, so its final-epoch error sat far above its best, while mixup at α = 32 beat the best dropout setting (14.1% best, 15.5% last) and mixup combined with dropout reached 10.9% at both points. Under white-box FGSM attacks on ImageNet, a mixup ResNet-101 had 75.2% top-1 error against ERM's 90.7%, but neither model resisted the iterative I-FGSM attack.

The CIFAR-10 ablation with PreAct ResNet-18 isolates the ingredients: standard mixup (all classes, random pairs) reached 4.24% against ERM's 5.53% at the same weight decay, nearest-neighbour partners 4.98%, mixing inputs without mixing labels 5.17–5.45%, label smoothing 5.25–5.34% and Gaussian input noise 5.53–7.16%. mixup preferred the lighter weight decay of 10^⁻⁴ while ERM preferred the heavier one, which the authors read as mixup supplying regularization of its own. And mixup is not free accuracy: on speech it hurt the small LeNet at both α = 0.1 and 0.2.

The paper is clear that the knob has a wrong end: push α too high and the model spends its capacity on blends that no real input resembles, and underfits. It also reports the effect as strongest for larger models and longer schedules — a regulariser earns its keep only where there is something to regularise, which is the same conditional that decided dropout's fate at the top of this ladder.

The mechanism is contested, and unusually the disagreement is about which of several real effects is the main one. Candidate accounts include vicinal risk minimisation (the paper's own framing — train on a neighbourhood around each example rather than the point), a form of label smoothing that happens to be input-dependent, and improved calibration, since a network trained on soft targets stops being uniformly overconfident. The calibration effect is measurable and was not the stated goal.

How it held up: it became a default rather than a trick. mixup and its successor CutMix are standard in strong vision training recipes, including the ones that made vision transformers competitive on ImageNet-scale data without extra pre-training — which is the problem that rung leaves open. It transferred badly to text in its literal form, because the convex combination of two token sequences is not a sequence; the workable adaptations blend hidden representations instead of inputs, which keeps the geometry and gives up the simplicity that made the original notable.

mixup became a standard ingredient of the modern training recipe, especially in vision. It's a default in many strong image-classification pipelines and part of the "bag of tricks" that squeezes extra accuracy out of ResNets and Vision Transformers — indeed, the heavy augmentation regimes that make data-hungry ViTs trainable lean on mixup and its relatives heavily. Its close cousin CutMix pastes a patch of one image into another and mixes labels by area; the two are frequently used together.

Conceptually, mixup elevated data augmentation from a bag of ad-hoc image tweaks (crop, flip, rotate) to a principled tool grounded in vicinal risk minimization — reshaping what counts as a training example rather than just perturbing pixels. It sits alongside other cheap, architecture-free regularizers like dropout in the standard toolkit, and its calibration and robustness benefits made it valuable wherever trustworthy confidence matters, not just raw accuracy. That so trivial an intervention — average two examples, average their labels — could reliably improve generalization was itself the lesson: sometimes the biggest lever is on the data, not the model.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Seven papers that turned "deeper is better in theory" into a network that actually converges. Every later architecture paper assumes all of them.

1. 1Dropout2014
2. 2Batch Normalization2015
3. 3Adam2015
4. 4ResNet2016
5. 5LayerNorm & RMSNorm2016
6. 6mixup2018
7. 7The Lottery Ticket Hypothesis2019

All 110 papers, in 19 ladders →