ResNet,
explained.

By 2015, depth was the frontier — more layers, more power. But the authors hit a wall that made no sense: a 56-layer plain network had higher error than a 20-layer one, and not just on test data — on the training data too.

That rules out overfitting, which would show up only at test time. The deeper network literally couldn't fit data the shallower one handled easily. The problem wasn't capacity; it was that very deep plain networks had become too hard to optimize. This was the degradation problem.

Here's the logic that cracked it. Suppose your best 20-layer network is great. A 56-layer network could match it exactly — just make the 36 extra layers do nothing, passing their input straight through. So more depth should never hurt; worst case, the extra layers learn to be invisible.

The trouble is that a stack of plain layers finds "do nothing" — the identity function — surprisingly hard to learn. So the authors changed the architecture to make doing nothing the easy, default option.

Their fix: add a shortcut that carries a block's input directly to its output, and let the layers add to it. Instead of computing a whole new output, the block computes output = input + F(input). The layers only have to learn F — the residual, the change to apply — not the entire mapping.

Now "do nothing" is trivial: the block just learns F = 0, and the shortcut passes the input through untouched. Extra depth can only help, never actively hurt.

Skip connections do two quiet, crucial things. They make each block's default behavior "do no harm," so optimization starts from a sane place. And they give gradients a clean highway straight back through the network during training — sidestepping the vanishing-gradient problem where the learning signal fades to nothing before it reaches the early layers of a very deep net.

ResNet won the ImageNet 2015 classification challenge and swept detection and localization, with a 152-layer network reaching around 3.6% top-5 error — better than human-level on that benchmark, and a huge leap over the previous year's winners.

The residual connection escaped computer vision and went everywhere. Look inside a Transformer — the architecture behind every modern LLM — and each block wraps its attention and feed-forward layers in exactly this output = input + sublayer(input) pattern. It's a big reason deep Transformers train stably at all.

ResNet's lesson is one of the most reused in all of deep learning: don't force a block to reinvent its input from scratch — hand it the input for free and let it learn only what to change. A one-line idea that made "deep" learning actually deep.