ResNet,
explained.

By 2015, depth was the frontier — more layers, more power. But the authors hit a wall that made no sense: a 56-layer plain network had higher error than a 20-layer one, and not just on test data — on the training data too.

That rules out overfitting, which would show up only at test time. The deeper network literally couldn't fit data the shallower one handled easily. The problem wasn't capacity; it was that very deep plain networks had become too hard to optimize. This was the degradation problem.

Here's the logic that cracked it. Suppose your best 20-layer network is great. A 56-layer network could match it exactly — just make the 36 extra layers do nothing, passing their input straight through. So more depth should never hurt; worst case, the extra layers learn to be invisible.

The trouble is that a stack of plain layers finds "do nothing" — the identity function — surprisingly hard to learn. So the authors changed the architecture to make doing nothing the easy, default option.

Their fix: add a shortcut that carries a block's input directly to its output, and let the layers add to it. Instead of computing a whole new output, the block computes output = input + F(input). The layers only have to learn F — the residual, the change to apply — not the entire mapping.

Now "do nothing" is trivial: the block just learns F = 0, and the shortcut passes the input through untouched. Extra depth can only help, never actively hurt.

Skip connections do two quiet, crucial things. They make each block's default behavior "do no harm," so optimization starts from a sane place. And they give gradients a clean highway straight back through the network during training — sidestepping the vanishing-gradient problem where the learning signal fades to nothing before it reaches the early layers of a very deep net.

The reason is one derivative. Backprop scales the gradient by each block's Jacobian; for y = x + F(x) that Jacobian is:

In a plain net the gradient is a bare product of the layer factors — a^L — and if each factor is below 1 it decays to nothing across a deep stack. The residual's +1 adds an identity path, so the gradient always contains a term that skips every layer untouched and arrives at full strength:

The gradient can't reach zero when one of its paths is the identity — depth stops being fatal. The runnable version below lets you watch it happen.

ResNet won the ImageNet 2015 classification challenge and swept detection and localization, with a 152-layer network reaching around 3.6% top-5 error — better than human-level on that benchmark, and a huge leap over the previous year's winners.

The residual connection escaped computer vision and went everywhere. Look inside a Transformer — the architecture behind every modern LLM — and each block wraps its attention and feed-forward layers in exactly this output = input + sublayer(input) pattern. It's a big reason deep Transformers train stably at all.

ResNet's lesson is one of the most reused in all of deep learning: don't force a block to reinvent its input from scratch — hand it the input for free and let it learn only what to change. A one-line idea that made "deep" learning actually deep.

The residual connection — output = F(x) + x — is one of the most quietly universal ideas in deep learning. Before ResNet, stacking more layers made networks worse, not better: gradients vanished on the long trip back, and deep networks couldn't even match shallower ones. The skip connection gave the gradient a clean highway straight through, so a 152-layer network became trainable where a 30-layer one had stalled.

The reframing was the genius part. Instead of asking each block to learn the full transformation it wants, ResNet asks it to learn only the residual — the small change from the input. If the best thing a layer can do is nothing, it just learns to push F(x) toward zero and passes the input through unchanged. "Do no harm" becomes the easy default, so adding depth stops hurting.

That idea did not stay in vision. The Transformer is built on residual connections around every attention and feed-forward block — without them, models with dozens or hundreds of layers would be untrainable. Every large language model you use today relies on the exact mechanism ResNet introduced. It is the structural reason "just add more layers" finally works.

The lasting lesson: when depth stops helping, the problem is often the flow of information and gradients, not capacity. Give signals a shortcut path and make the identity the easy default, and you can train networks far deeper than anyone thought possible. ResNet turned depth from a liability into the field's main lever.