Vision Transformer,
explained.

Every strong vision model between rung 1 and this one was convolutional, and the reason was principled: a convolution assumes that nearby pixels are related and that a cat is a cat wherever it appears. Those assumptions are true, they are baked into the architecture rather than learned, and they are why a CNN can be trained on a hundred thousand images at all.

ViT's change is to remove them. Cut the image into fixed-size patches, flatten each one, project it to a vector, and hand the sequence to an unmodified transformer encoder. At 16×16 patches on a 224-pixel image that is 196 tokens, plus one learned [class] token whose final representation is classified. Position is supplied by a learned embedding, which means the model is told that patches have an order rather than being built to know it. Nothing in the architecture knows that patch 15 is next to patch 16.

Stated in rung 1's units, the result is a genuine trade rather than a win. Trained on ImageNet-scale data alone, ViT is worse than a comparable ResNet — the paper reports this directly. Pre-trained on one to two orders of magnitude more images and then transferred, it is better, and it keeps improving as data grows where the CNN's curve flattens. So the delta is not "attention beats convolution". It is: the convolutional prior is worth a large amount of data, and once you have more data than that, it is a ceiling.

Since AlexNet in 2012, convolutional neural networks owned computer vision. Convolutions bake in assumptions that fit images beautifully: locality (nearby pixels relate), translation equivariance (a cat is a cat wherever it appears), and a hierarchy from edges to shapes to objects. These "inductive biases" are why CNNs learn efficiently from modest data.

Meanwhile Transformers had taken over NLP by throwing out recurrence in favor of pure attention. The obvious question hung in the air — could attention replace convolution too? — but the consensus was no: images have millions of pixels, and letting every pixel attend to every other is computationally impossible, and Transformers lack the vision-friendly biases CNNs have baked in.

The workaround is almost cheeky. You can't treat each pixel as a token — too many. So don't. Cut the image into a grid of fixed-size patches (say 16×16 pixels each). A 224×224 image becomes just 196 patches. Now treat each patch as a "word" and the image as a short sentence of a couple hundred tokens — a length Transformers handle with ease.

That's the entire trick. The paper's title says it outright: An Image is Worth 16×16 Words. Once an image is a sequence of patch-tokens, you can feed it to the same Transformer encoder that reads text, with essentially no vision-specific changes.

Each patch is a little block of pixels. Flatten those pixels into a vector and pass it through a single linear projection — a learned matrix that maps the raw pixels into a patch embedding, the visual equivalent of a word embedding. Because a Transformer has no built-in sense of order, you also add a positional embedding to each patch so the model knows where in the grid it came from.

The result is a sequence of embeddings — exactly the input a Transformer expects. From here, standard multi-head self-attention lets every patch look at every other patch, so the model can relate the top-left of an image to the bottom-right in a single layer, something a CNN only achieves after stacking many layers.

The counting is exact. Split an H×W image into P×P patches and you get a fixed grid, each patch a flattened pixel vector — for the canonical ViT-Base/16 on a 224×224 image:

But that grid size isn't free, because self-attention is O(N²) in the token count. Want finer detail from smaller patches? Every halving of the patch side quadruples the tokens and so 16×es the attention compute:

Patch size is the dial that trades spatial resolution against a quadratic compute bill — the single most important knob in a ViT.

How do you get a single label out of a bag of patch representations? ViT borrows BERT's trick: prepend one extra learnable token — the [class] token — to the sequence. It carries no pixels of its own; it's a blank slot whose job is to gather information.

As the Transformer's layers mix information through attention, this token accumulates a summary of the whole image. After the final layer, you read off its vector and attach a small classification head to predict the label. One token in, one summary out — clean and architecture-agnostic.

There's no free lunch. By discarding convolution, ViT also discards its helpful biases. It doesn't assume nearby patches are related or that objects can move — it has to learn all of that from examples. On a mid-sized dataset like ImageNet alone, ViT actually loses to a comparable CNN, precisely because the CNN's baked-in assumptions are a head start.

The magic appears at scale. Pre-train ViT on a truly enormous dataset — hundreds of millions of images (JFT-300M) — and it not only catches up but surpasses the best CNNs. With enough data, the model learns the right visual structure itself, and its freedom from rigid biases becomes an advantage rather than a handicap.

Pre-trained on JFT-300M and fine-tuned on standard benchmarks, ViT matched or beat state-of-the-art CNNs (like the best ResNets) on ImageNet and friends — at lower pre-training compute. It proved, concretely, that convolution was not a requirement for top-tier vision; it was one option among others.

An intriguing bonus: analysis of ViT's attention showed that even in early layers, some heads attend globally across the whole image — learning a kind of long-range "receptive field" that CNNs only build up gradually. The model discovered its own way to see.

ViT's dependence on gigantic pre-training datasets was a real barrier — few teams have JFT-300M. The plain design also struggles with high-resolution images and dense tasks like detection and segmentation, where the fixed patch grid is coarse. And self-attention over many patches is quadratic in cost, which bites as images (and patch counts) grow.

Follow-up work rushed to fix these: DeiT showed ViT could be trained data-efficiently with distillation; hierarchical designs like the Swin Transformer reintroduced locality and multi-scale structure for detection and segmentation. Pure ViT was a proof of concept more than a final answer.

ViT broke the assumption that vision needs its own special architecture, and in doing so it helped unify AI around one backbone: the Transformer now processes text, images, audio, and video with the same core machinery. That convergence is why multimodal models feel natural — everything is just a sequence of tokens.

Its fingerprints are everywhere downstream: CLIP's image encoder, the vision half of multimodal LLMs, and modern image and video generators all lean on Transformer-over-patches. When a single model can look at a picture and read a caption in the same representation space, it's because ViT showed a patch could be a token.

The data appetite is stated as a finding rather than buried — the paper's own framing is that large-scale pre-training is what makes the approach work, and that below that threshold the missing inductive bias costs you. It also notes the practical consequence: attention over patches is quadratic in the number of patches, so higher resolution is expensive in a way convolution is not.

The data-hunger claim was then substantially weakened by better training, which is the most useful thing that happened to this paper. Touvron and colleagues' DeiT recipe made a ViT competitive on ImageNet-1k alone using heavy augmentation — mixup and CutMix among them — plus distillation from a convolutional teacher. Read together, the two results say the original ceiling was partly an artefact of the training recipe rather than a property of the architecture, which is a caution worth carrying to every other "needs enormous data" claim on this site.

How it held up: the patch-plus-transformer backbone is now the default for anything that has to put images and text in the same representation — it is the image tower in CLIP and the visual encoder feeding LLaVA. Pure global attention did not win unchallenged: hierarchical designs such as Swin reintroduced locality and multi-scale windows and remain strong for dense prediction, which suggests the convolutional prior was right about something even where it was not required. And the [class] token itself is largely gone — pooling over patch tokens works as well or better, and rung 5's features come from the patch tokens directly.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

From the network that started the deep-learning era to one you can segment anything with — and the shift from labels to self-supervision in between.

1. 1AlexNet2012
2. 2Vision Transformer2021
3. 3SimCLR2020
4. 4Masked Autoencoders2022
5. 5DINOv22023
6. 6Segment Anything2023

All 110 papers, in 19 ladders →