ResNet,
explained.

Everything below this rung makes a network of a given depth train better: dropout generalises it, batch norm conditions it, Adam steps it sensibly. ResNet changes what depths are reachable at all. The problem it names is not overfitting, and the paper is careful to establish that: stacking more layers made the network worse on the training set as well. A model that cannot fit the data it has seen is not overfitting; it is failing to optimise.

The argument is almost embarrassingly simple. A deeper network contains a shallower one as a special case — set the extra layers to the identity and you have exactly the shallow model — so a deeper network should never be worse. That it was worse means the optimiser could not find the identity. So make the identity the default: instead of asking a block to learn its output H(x), ask it to learn the residual F(x), and add the input back: H(x) = F(x) + x. Now doing nothing costs zero effort, and every layer only has to learn its own correction.

Stated in units the ladder can carry: before this, useful plain networks topped out around 20-odd layers, with deeper ones degrading. The paper trains 152 layers to win ImageNet and reports a 1,000-layer variant as trainable. The skip adds no parameters when the shapes match — it is an addition, not a layer — which is why it is the cheapest structural idea on this ladder by a wide margin.

By 2015, depth was the frontier — more layers, more power. But the authors hit a wall that made no sense: a 56-layer plain network had higher error than a 20-layer one, and not just on test data — on the training data too.

That rules out overfitting, which would show up only at test time. The deeper network literally couldn't fit data the shallower one handled easily. The problem wasn't capacity; it was that very deep plain networks had become too hard to optimize. This was the degradation problem.

Here's the logic that cracked it. Suppose your best 20-layer network is great. A 56-layer network could match it exactly — just make the 36 extra layers do nothing, passing their input straight through. So more depth should never hurt; worst case, the extra layers learn to be invisible.

The trouble is that a stack of plain layers finds "do nothing" — the identity function — surprisingly hard to learn. So the authors changed the architecture to make doing nothing the easy, default option.

Their fix: add a shortcut that carries a block's input directly to its output, and let the layers add to it. Instead of computing a whole new output, the block computes output = input + F(input). The layers only have to learn F — the residual, the change to apply — not the entire mapping.

Now "do nothing" is trivial: the block just learns F = 0, and the shortcut passes the input through untouched. Extra depth can only help, never actively hurt.

Skip connections do two quiet, crucial things. They make each block's default behavior "do no harm," so optimization starts from a sane place. And they give gradients a clean highway straight back through the network during training — sidestepping the vanishing-gradient problem where the learning signal fades to nothing before it reaches the early layers of a very deep net.

The reason is one derivative. Backprop scales the gradient by each block's Jacobian; for y = x + F(x) that Jacobian is:

In a plain net the gradient is a bare product of the layer factors — a^L — and if each factor is below 1 it decays to nothing across a deep stack. The residual's +1 adds an identity path, so the gradient always contains a term that skips every layer untouched and arrives at full strength:

The gradient can't reach zero when one of its paths is the identity — depth stops being fatal. The runnable version below lets you watch it happen.

ResNet won the ImageNet 2015 classification challenge and swept detection and localization, with a 152-layer network reaching around 3.6% top-5 error — better than human-level on that benchmark, and a huge leap over the previous year's winners.

The residual connection escaped computer vision and went everywhere. Look inside a Transformer — the architecture behind every modern LLM — and each block wraps its attention and feed-forward layers in exactly this output = input + sublayer(input) pattern. It's a big reason deep Transformers train stably at all.

ResNet's lesson is one of the most reused in all of deep learning: don't force a block to reinvent its input from scratch — hand it the input for free and let it learn only what to change. A one-line idea that made "deep" learning actually deep.

The paper reports a negative result it did not have to: the very deepest variant it trains, past a thousand layers, performs worse than the 152-layer model despite training successfully, and the authors attribute it to overfitting on a dataset that small rather than claiming depth scales without end. Naming the point where your own curve turns over is rarer than it should be.

The stated mechanism is again the contested part. The paper's framing is that residuals are easier to learn than full mappings. Later work offered different accounts — that skips keep gradients flowing to early layers almost undiminished, and, in Veit and colleagues' reading, that a residual network behaves less like one very deep network than like an ensemble of many shallower paths, since dropping individual blocks at test time barely hurts it. All three accounts predict that it works; they disagree about what it is.

How it held up: it is the most universally adopted idea on this ladder. Every transformer block on this site is a residual block — attention and the feed-forward network are both written as x plus something — and the "residual stream" that interpretability work reads, on the transformer ladder's top rung, is exactly this addition viewed as a channel that every layer reads from and writes to. Rung 5 is partly a story about where the normaliser sits relative to this skip.

The residual connection — output = F(x) + x — is one of the most quietly universal ideas in deep learning. Before ResNet, stacking more layers made networks worse, not better: gradients vanished on the long trip back, and deep networks couldn't even match shallower ones. The skip connection gave the gradient a clean highway straight through, so a 152-layer network became trainable where a 30-layer one had stalled.

The reframing was the genius part. Instead of asking each block to learn the full transformation it wants, ResNet asks it to learn only the residual — the small change from the input. If the best thing a layer can do is nothing, it just learns to push F(x) toward zero and passes the input through unchanged. "Do no harm" becomes the easy default, so adding depth stops hurting.

That idea did not stay in vision. The Transformer is built on residual connections around every attention and feed-forward block — without them, models with dozens or hundreds of layers would be untrainable. Every large language model you use today relies on the exact mechanism ResNet introduced. It is the structural reason "just add more layers" finally works.

The lasting lesson: when depth stops helping, the problem is often the flow of information and gradients, not capacity. Give signals a shortcut path and make the identity the easy default, and you can train networks far deeper than anyone thought possible. ResNet turned depth from a liability into the field's main lever.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

Seven papers that turned "deeper is better in theory" into a network that actually converges. Every later architecture paper assumes all of them.

1. 1Dropout2014
2. 2Batch Normalization2015
3. 3Adam2015
4. 4ResNet2016
5. 5LayerNorm & RMSNorm2016
6. 6mixup2018
7. 7The Lottery Ticket Hypothesis2019

All 110 papers, in 19 ladders →