AlexNet,
explained.

Rung 1, so the comparison is against the practice it ended. Vision before this was a two-stage craft: a human designed a feature extractor — SIFT, HOG, a bag of visual words — and a classifier was trained on top of the numbers it produced. The features were the intellectual content, and they were fixed before any data was seen. Improving a system meant designing a better descriptor.

The change is to delete the first stage. Learn the features and the classifier together, from pixels, with one loss. That idea was decades old and had never been decisive at scale; what makes this the rung is that three things arrived at once to make it work — a dataset large enough to support it, GPUs fast enough to train on it, and a handful of choices that kept a deep network trainable.

The units to carry up the ladder: eight learned layers (five convolutional, three fully connected), about 60M parameters, trained across two GPUs because the model did not fit on one, and an ImageNet top-5 error around 15% against roughly 26% for the best hand-designed entry — a margin large enough that it was not arguable. The choices that made it train are the ones to remember, because two of them are rungs on the training ladder: ReLU instead of a saturating activation, which converges several times faster because its gradient does not vanish for positive inputs; dropout on the fully-connected layers; and aggressive data augmentation.

For decades, computer vision meant hand-engineered features. Experts designed detectors — SIFT, HOG, edge and corner descriptors — to extract useful patterns from images, then fed those to a classifier like an SVM. Progress came from human cleverness in feature design, one careful engineering step at a time.

Neural networks had been around since the 1980s, and convolutional nets even worked on small problems like digit recognition. But on real, messy, high-resolution images they were considered impractical — too data-hungry, too slow, too prone to overfit. The consensus was that deep learning was a dead end for serious vision. AlexNet's job, though nobody framed it that way, was to demolish that consensus.

ImageNet was the arena: a competition to classify images into 1,000 categories, drawing from over a million labeled photos. In 2012, the leading approaches — refined hand-engineered pipelines — hovered around a top-5 error rate of ~26%. AlexNet posted ~15%.

An eleven-point drop in a mature benchmark is not an improvement; it's an earthquake. The margin was so far beyond the runner-up that it couldn't be dismissed as a lucky tweak. Within a year, essentially the entire vision community pivoted to deep convolutional networks. This one result is routinely cited as the spark of the modern deep learning era.

AlexNet is a deep convolutional neural network: five convolutional layers that learn a hierarchy of visual features — early layers catching edges and colors, later layers assembling textures, parts, and whole objects — followed by three fully-connected layers that map those features to the 1,000 class scores. Pooling layers shrink the spatial size along the way. About 60 million parameters in total.

The spatial sizes aren't magic either — every feature map's dimension follows one formula, from the input image to the kernel size, stride, and padding:

The famous 11×11 stride-4 first layer turns a 227×227 image into a 55×55 grid of 96 features — computed, not guessed.

The design itself wasn't wildly novel — convolution and pooling were old. What made it work at this depth and scale was a bundle of practical choices, each solving a problem that had blocked deep nets before.

Earlier networks used saturating activations like tanh or sigmoid, whose curves flatten out for large inputs — and where the curve is flat, the gradient is nearly zero, so learning crawls. AlexNet used the ReLU: f(x) = max(0, x). Dead simple, and it doesn't saturate on the positive side, so gradients stay strong.

The payoff was several times faster training. For a network this deep, that speedup wasn't a nicety — it was the difference between a training run finishing and not. ReLU has been the default activation for most deep networks ever since, a lasting contribution from this one paper.

Sixty million parameters can memorize a training set instead of learning to generalize. AlexNet fought this on two fronts. Dropout randomly switches off a fraction of neurons on each training pass, so the network can't lean on any single neuron or path — it must learn redundant, robust features, like a team that keeps working when random members are out sick.

Data augmentation multiplies the training data for free: random crops, horizontal flips, and color jitter turn one labeled image into many plausible variants. The model sees a cat shifted, mirrored, and re-lit, and learns that none of those change the label — teaching invariance that hand-labeled data alone couldn't. Both tricks are now standard practice.

None of this would have been trainable on CPUs in reasonable time. The authors implemented the whole network on GPUs, whose thousands of cores devour the parallel matrix math that neural networks are made of. The model was even split across two GPUs to fit within the memory of the day, with a clever cross-communication scheme.

That practical decision may be AlexNet's most consequential legacy. It demonstrated, concretely, that GPUs turn deep learning from theoretically-possible into actually-doable — kicking off the hardware-and-data flywheel that has driven the field ever since. The line from AlexNet's two GPUs to today's massive training clusters is direct.

By modern standards AlexNet is crude: large early filters, a somewhat ad-hoc structure, and a design shaped partly by 2012 GPU memory limits. Its successors quickly improved on it — VGG went deeper with small filters, GoogLeNet added inception modules, and ResNet introduced skip connections that made truly deep networks trainable.

It also depended on a specific stack aligning at once: a big labeled dataset, GPUs powerful enough, and the right regularization. Take any one away and it wouldn't have worked — which is part of why the moment arrived exactly when it did, and not years earlier.

AlexNet is the "big bang" of modern AI. It proved that a general recipe — a deep network, a lot of data, and enough compute — could beat decades of specialized human engineering, and it did so on a benchmark everyone respected. The result triggered a stampede into deep learning that never reversed, seeding the labs, funding, and talent behind everything that followed.

The specific tools it popularized — ReLU, dropout, augmentation, GPU training — remain in nearly every deep learning system today. And its central lesson, that learned features beat hand-designed ones given enough scale, is the same conviction that later drove the scaling of language models. Every model in this library owes something to the door AlexNet kicked open.

The paper's most interesting admission is a prediction. It states plainly that depth is what matters — removing a single convolutional layer degrades results measurably — and that performance was limited by how much compute and data were available, not by the method, so it should be expected to improve with faster hardware and larger datasets. That is the scaling intuition, stated a decade before the curves that made it quantitative.

Not everything in it survived. Local response normalisation, included as a stabiliser, was superseded by batch normalisation and dropped. The two-GPU split with its restricted cross-connections was a memory workaround, not a design, and the authors say so. The overlapping-pooling and specific kernel sizes were tuned rather than derived.

How it held up: the method won so completely that the question it answered stopped being asked. Every rung above this one takes learned features for granted. ReLU, dropout, heavy augmentation and training on accelerators are all still defaults. What did not hold is the assumption underneath the architecture — that a vision model must be convolutional, because locality and translation-equivariance are how images work. Rung 2 removes convolution entirely and finds that the inductive bias was a substitute for data rather than a requirement.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

From the network that started the deep-learning era to one you can segment anything with — and the shift from labels to self-supervision in between.

1. 1AlexNet2012
2. 2Vision Transformer2021
3. 3SimCLR2020
4. 4Masked Autoencoders2022
5. 5DINOv22023
6. 6Segment Anything2023

All 110 papers, in 19 ladders →