Vision Transformer,
explained.

Since AlexNet in 2012, convolutional neural networks owned computer vision. Convolutions bake in assumptions that fit images beautifully: locality (nearby pixels relate), translation equivariance (a cat is a cat wherever it appears), and a hierarchy from edges to shapes to objects. These "inductive biases" are why CNNs learn efficiently from modest data.

Meanwhile Transformers had taken over NLP by throwing out recurrence in favor of pure attention. The obvious question hung in the air — could attention replace convolution too? — but the consensus was no: images have millions of pixels, and letting every pixel attend to every other is computationally impossible, and Transformers lack the vision-friendly biases CNNs have baked in.

The workaround is almost cheeky. You can't treat each pixel as a token — too many. So don't. Cut the image into a grid of fixed-size patches (say 16×16 pixels each). A 224×224 image becomes just 196 patches. Now treat each patch as a "word" and the image as a short sentence of a couple hundred tokens — a length Transformers handle with ease.

That's the entire trick. The paper's title says it outright: An Image is Worth 16×16 Words. Once an image is a sequence of patch-tokens, you can feed it to the same Transformer encoder that reads text, with essentially no vision-specific changes.

Each patch is a little block of pixels. Flatten those pixels into a vector and pass it through a single linear projection — a learned matrix that maps the raw pixels into a patch embedding, the visual equivalent of a word embedding. Because a Transformer has no built-in sense of order, you also add a positional embedding to each patch so the model knows where in the grid it came from.

The result is a sequence of embeddings — exactly the input a Transformer expects. From here, standard multi-head self-attention lets every patch look at every other patch, so the model can relate the top-left of an image to the bottom-right in a single layer, something a CNN only achieves after stacking many layers.

How do you get a single label out of a bag of patch representations? ViT borrows BERT's trick: prepend one extra learnable token — the [class] token — to the sequence. It carries no pixels of its own; it's a blank slot whose job is to gather information.

As the Transformer's layers mix information through attention, this token accumulates a summary of the whole image. After the final layer, you read off its vector and attach a small classification head to predict the label. One token in, one summary out — clean and architecture-agnostic.

There's no free lunch. By discarding convolution, ViT also discards its helpful biases. It doesn't assume nearby patches are related or that objects can move — it has to learn all of that from examples. On a mid-sized dataset like ImageNet alone, ViT actually loses to a comparable CNN, precisely because the CNN's baked-in assumptions are a head start.

The magic appears at scale. Pre-train ViT on a truly enormous dataset — hundreds of millions of images (JFT-300M) — and it not only catches up but surpasses the best CNNs. With enough data, the model learns the right visual structure itself, and its freedom from rigid biases becomes an advantage rather than a handicap.

Pre-trained on JFT-300M and fine-tuned on standard benchmarks, ViT matched or beat state-of-the-art CNNs (like the best ResNets) on ImageNet and friends — at lower pre-training compute. It proved, concretely, that convolution was not a requirement for top-tier vision; it was one option among others.

An intriguing bonus: analysis of ViT's attention showed that even in early layers, some heads attend globally across the whole image — learning a kind of long-range "receptive field" that CNNs only build up gradually. The model discovered its own way to see.

ViT's dependence on gigantic pre-training datasets was a real barrier — few teams have JFT-300M. The plain design also struggles with high-resolution images and dense tasks like detection and segmentation, where the fixed patch grid is coarse. And self-attention over many patches is quadratic in cost, which bites as images (and patch counts) grow.

Follow-up work rushed to fix these: DeiT showed ViT could be trained data-efficiently with distillation; hierarchical designs like the Swin Transformer reintroduced locality and multi-scale structure for detection and segmentation. Pure ViT was a proof of concept more than a final answer.

ViT broke the assumption that vision needs its own special architecture, and in doing so it helped unify AI around one backbone: the Transformer now processes text, images, audio, and video with the same core machinery. That convergence is why multimodal models feel natural — everything is just a sequence of tokens.

Its fingerprints are everywhere downstream: CLIP's image encoder, the vision half of multimodal LLMs, and modern image and video generators all lean on Transformer-over-patches. When a single model can look at a picture and read a caption in the same representation space, it's because ViT showed a patch could be a token.