Vision Transformer,
explained.

Every strong vision model between rung 1 and this one was convolutional, and the reason was principled: a convolution assumes that nearby pixels are related and that a cat is a cat wherever it appears. Those assumptions are true, they are baked into the architecture rather than learned, and they are why a CNN can be trained on a hundred thousand images at all.

ViT's change is to remove them. Cut the image into fixed-size patches, flatten each one, project it to a vector, and hand the sequence to an unmodified transformer encoder. At 16×16 patches on a 224-pixel image that is 196 tokens, plus one learned [class] token whose final representation is classified. Position is supplied by a learned embedding, which means the model is told that patches have an order rather than being built to know it. Nothing in the architecture knows that patch 15 is next to patch 16.

Stated in rung 1's units, the result is a genuine trade rather than a win. Trained on ImageNet-scale data alone, ViT is worse than a comparable ResNet — the paper reports this directly. Pre-trained on one to two orders of magnitude more images and then transferred, it is better, and it keeps improving as data grows where the CNN's curve flattens. So the delta is not "attention beats convolution". It is: the convolutional prior is worth a large amount of data, and once you have more data than that, it is a ceiling.

Since AlexNet in 2012, convolutional neural networks owned computer vision. Convolutions bake in assumptions that fit images beautifully: locality (nearby pixels relate), translation equivariance (a cat is a cat wherever it appears), and a hierarchy from edges to shapes to objects. These "inductive biases" are why CNNs learn efficiently from modest data.

Meanwhile Transformers had taken over NLP by throwing out recurrence in favor of pure attention. The obvious question hung in the air — could attention replace convolution too? — but the consensus was no: images have millions of pixels, and letting every pixel attend to every other is computationally impossible, and Transformers lack the vision-friendly biases CNNs have baked in.

The workaround is almost cheeky. You can't treat each pixel as a token — too many. So don't. Cut the image into a grid of fixed-size patches (say 16×16 pixels each). A 224×224 image becomes just 196 patches. Now treat each patch as a "word" and the image as a short sentence of a couple hundred tokens — a length Transformers handle with ease.

That's the entire trick. The paper's title says it outright: An Image is Worth 16×16 Words. Once an image is a sequence of patch-tokens, you can feed it to the same Transformer encoder that reads text, with essentially no vision-specific changes.

Each patch is a little block of pixels. Flatten those pixels into a vector and pass it through a single linear projection — a learned matrix that maps the raw pixels into a patch embedding, the visual equivalent of a word embedding. Because a Transformer has no built-in sense of order, you also add a positional embedding to each patch so the model knows where in the grid it came from.

The result is a sequence of embeddings — exactly the input a Transformer expects. From here, standard multi-head self-attention lets every patch look at every other patch, so the model can relate the top-left of an image to the bottom-right in a single layer, something a CNN only achieves after stacking many layers.

The counting is exact. Split an H×W image into P×P patches and you get a fixed grid, each patch a flattened pixel vector — for the canonical ViT-Base/16 on a 224×224 image:

But that grid size isn't free, because self-attention is O(N²) in the token count. Want finer detail from smaller patches? Every halving of the patch side quadruples the tokens and so 16×es the attention compute:

Patch size is the dial that trades spatial resolution against a quadratic compute bill — the single most important knob in a ViT.

How do you get a single label out of a bag of patch representations? ViT borrows BERT's trick: prepend one extra learnable token — the [class] token — to the sequence. It carries no pixels of its own; it's a blank slot whose job is to gather information.

As the Transformer's layers mix information through attention, this token accumulates a summary of the whole image. After the final layer, you read off its vector and attach a small classification head to predict the label. One token in, one summary out — clean and architecture-agnostic.

There's no free lunch. By discarding convolution, ViT also discards its helpful biases. It doesn't assume nearby patches are related or that objects can move — it has to learn all of that from examples. On a mid-sized dataset like ImageNet alone, ViT actually loses to a comparable CNN, precisely because the CNN's baked-in assumptions are a head start.

The magic appears at scale. Pre-train ViT on a truly enormous dataset — hundreds of millions of images (JFT-300M) — and it not only catches up but surpasses the best CNNs. With enough data, the model learns the right visual structure itself, and its freedom from rigid biases becomes an advantage rather than a handicap.

ViT's dependence on gigantic pre-training datasets was a real barrier — few teams have JFT-300M. The plain design also struggles with high-resolution images and dense tasks like detection and segmentation, where the fixed patch grid is coarse. And self-attention over many patches is quadratic in cost, which bites as images (and patch counts) grow.

Follow-up work rushed to fix these: DeiT showed ViT could be trained data-efficiently with distillation; hierarchical designs like the Swin Transformer reintroduced locality and multi-scale structure for detection and segmentation. Pure ViT was a proof of concept more than a final answer.

ViT broke the assumption that vision needs its own special architecture, and in doing so it helped unify AI around one backbone: the Transformer now processes text, images, audio, and video with the same core machinery. That convergence is why multimodal models feel natural — everything is just a sequence of tokens.

Its fingerprints are everywhere downstream: CLIP's image encoder, the vision half of multimodal LLMs, and modern image and video generators all lean on Transformer-over-patches. When a single model can look at a picture and read a caption in the same representation space, it's because ViT showed a patch could be a token.

ViT follows the original Transformer encoder as closely as it can; almost everything specific to images happens before the first attention layer and after the last one. The steps below use the base model at 224×224 resolution with 16×16 patches.

1. Cut the image into patches. An H×W image with C channels becomes N = HW/P² flattened patches, each a vector of P²·C raw pixel values. At 224×224 with P = 16, that is 196 patches of 16·16·3 = 768 numbers each.
2. Project each patch. One learned matrix E maps every flattened patch to a vector of width D. The same E is used for every patch, so this is a convolution with kernel and stride equal to the patch size. D is fixed for the whole network: 768 for the base model.
3. Prepend a [class] embedding. A learnable vector is placed at the front of the sequence, making N + 1 = 197 tokens. It carries no pixels; the final state at its position is the image representation.
4. Add position embeddings. A learned table of N + 1 vectors is added to the sequence, one per position in raster order. The model is not told that patch 15 sits next to patch 16 or above patch 31; it learns that from data. The authors tried 2D-aware variants and saw no significant gain over the plain 1D table.
5. Run L encoder blocks. Each block applies layer normalization, multi-head self-attention and a residual connection, then layer normalization, a two-layer MLP with a GELU in between and a second residual connection. Normalization comes before each sub-layer, which is the pre-norm arrangement that trains more stably than the original post-norm layout. The MLP hidden width is 4D.
6. Read out the class token. The final state of the [class] position is normalized and passed to a classification head. At pre-training time the head is an MLP with one hidden layer; at fine-tuning time it is replaced by a single zero-initialized linear layer with one output per downstream class.

The three standard sizes

Base has 12 layers, width 768, MLP width 3,072, 12 heads and 86 million parameters. Large has 24 layers, width 1,024, MLP width 4,096, 16 heads and 307 million. Huge has 32 layers, width 1,280, MLP width 5,120, 16 heads and 632 million. A name such as ViT-L/16 means the Large model with 16×16 patches; ViT-H/14 uses 14×14 patches, so it sees longer sequences. A hybrid variant feeds feature maps from a ResNet into the same encoder instead of raw patches.

Training and fine-tuning

Pre-training uses Adam with a batch size of 4,096, a weight decay of 0.1, which the authors found helped transfer for every model, and a linear warmup and decay, at 224×224 resolution on ImageNet, ImageNet-21k (about 14 million images) or JFT-300M (about 300 million). Fine-tuning uses SGD with momentum and a batch size of 512, usually at a higher resolution: 384 by default, and 512 or 518 for the headline ImageNet results. The patch size stays fixed, so a higher resolution means a longer sequence. At 384 with P = 16 there are 576 patches plus the class token, 577 tokens in all.

The pre-trained position embeddings no longer match a longer sequence, so they are resized by 2D interpolation according to each patch's location in the original image. This step and the patch extraction are the only places the model is handed anything about the 2D structure of images.

Inspecting the trained model shows what the missing prior turned into. Some attention heads already span most of the image in the lowest layers, a global view a CNN builds only after many layers, while other heads stay local; the average attention distance grows with depth.

• xp^i — the i-th flattened patch, a vector of P²·C pixel values; E is the shared linear projection to width D.
• xclass, Epos — the learnable class embedding and the position table, one row per token including the class token.
• MSA, MLP, LN — multi-head self-attention, a two-layer MLP with GELU, and layer normalization. Normalization is applied before each sub-layer and the residual is added after it.
• zL^0 — the final state at position 0, the class token, which becomes the image representation y fed to the classification head.

For ViT-B/16 at 224×224, E has 768 × 768 = 589,824 entries and E_pos has 197 × 768 = 151,296. Attention compares every token with every other, so its score matrix has (N + 1)² entries: 38,809 at 224 resolution and 332,929 at 384, about 8.6 times more.

The paper pre-trains on ImageNet-1k, ImageNet-21k or JFT-300M, fine-tunes on ImageNet, CIFAR, Oxford Pets and Flowers, and the 19-task VTAB suite, and compares against BiT-L (a large ResNet pre-trained on JFT) and Noisy Student (an EfficientNet trained with extra unlabeled data). Cost is counted in TPUv3 core-days of pre-training.

ViT-H/14 matched or beat both baselines while using roughly a quarter of BiT-L's pre-training compute and a fifth of Noisy Student's. The authors caution that optimizer, schedule and weight decay also change efficiency, so part of the gap is not architecture alone.

Ablations

• Dataset size. Pre-trained on ImageNet-1k alone, ViT-Large did worse than ViT-Base and BiT beat both; on ImageNet-21k the two sizes were similar; only with JFT-300M did the larger model pay off. On random JFT subsets of 9, 30, 90 and 300 million images, ViT-B/32 did much worse than a ResNet50 at 9 million and better at 90 million and above.
• Compute. In a controlled study on JFT, ViT reached the same accuracy as ResNets with roughly 2 to 4 times less compute. Hybrids helped slightly at small budgets and the difference vanished at larger ones. ViT did not saturate in the range tried.
• Position information. Giving the model some position embedding mattered; 1D, 2D and relative variants differed little.
• Class token. Global average pooling over patch outputs worked as well as the class token once the learning rate was retuned.
• Self-supervision. A preliminary masked-patch objective brought ViT-B/16 to 79.9 percent on ImageNet, 2 points above training from scratch and 4 below supervised pre-training.

The data appetite is stated as a finding rather than buried — the paper's own framing is that large-scale pre-training is what makes the approach work, and that below that threshold the missing inductive bias costs you. It also notes the practical consequence: attention over patches is quadratic in the number of patches, so higher resolution is expensive in a way convolution is not.

The data-hunger claim was then substantially weakened by better training, which is the most useful thing that happened to this paper. Touvron and colleagues' DeiT recipe made a ViT competitive on ImageNet-1k alone using heavy augmentation — mixup and CutMix among them — plus distillation from a convolutional teacher. Read together, the two results say the original ceiling was partly an artefact of the training recipe rather than a property of the architecture, which is a caution worth carrying to every other "needs enormous data" claim on this site.

How it held up: the patch-plus-transformer backbone is now the default for anything that has to put images and text in the same representation — it is the image tower in CLIP and the visual encoder feeding LLaVA. Pure global attention did not win unchallenged: hierarchical designs such as Swin reintroduced locality and multi-scale windows and remain strong for dense prediction, which suggests the convolutional prior was right about something even where it was not required. And the [class] token itself is largely gone — pooling over patch tokens works as well or better, and rung 5's features combine the class token with pooled patch tokens.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

From the network that started the deep-learning era to one you can segment anything with — and the shift from labels to self-supervision in between.

1. 1AlexNet2012
2. 2Vision Transformer2021
3. 3SimCLR2020
4. 4Masked Autoencoders2022
5. 5DINOv22023
6. 6Segment Anything2023

All 110 papers, in 19 ladders →