CLIP,
explained.

Rung 1, so the comparison is against supervised image classification. A classifier is trained against a fixed list of categories, and that list is the ceiling on what it can ever say — recognising a new kind of thing means new labelled data and a new output layer. The labels are also expensive, which caps the data.

The change is to supervise with language instead of labels. Train an image encoder and a text encoder together on image-caption pairs so that a picture and its own caption land close together in one shared space, and mismatched pairs land apart — the contrastive objective from the vision ladder, with a caption in place of a second crop. The supervision now comes from text that already existed alongside the images, so the training set is roughly 400 million pairs scraped from the web rather than a hand-annotated corpus.

What falls out is the property that made it famous. Because the "labels" are arbitrary sentences, classification becomes a retrieval: write out candidate class names as short phrases, encode them, and pick whichever is nearest the image. You can classify into categories that were never a training target, define new ones at inference time by typing them, and — stated in units — match a strong supervised model on ImageNet without training on ImageNet at all. The fixed output layer is gone, replaced by a text encoder.

Before CLIP, a vision model was trained on a fixed list of labels — say, the 1,000 categories of ImageNet — and could only ever recognize those. Ask it about anything outside its list and it was helpless. Worse, building it meant an army of humans hand-labeling millions of images into rigid buckets.

Meanwhile language models understood rich, open-ended text. The two never met. What if a model could learn about images the way a person browsing the web does — from pictures paired with the captions people wrote for them?

CLIP — Contrastive Language-Image Pre-training — trains two encoders at once: one that turns an image into a vector, one that turns text into a vector. The goal is to align them so that a picture and its matching description land at nearly the same spot in a single shared embedding space.

Once images and words live on the same map, you can measure how well any picture matches any sentence just by checking how close their vectors are. Pixels and language become directly comparable.

Take a batch of image-caption pairs. Embed every image and every caption. Now train so each image is closest to its own caption and far from every other caption in the batch — pull the true pairs together, push the impostors apart.

Done across 400 million image-caption pairs scraped from the web — natural-language supervision, no hand-labeling — the two encoders converge into one aligned space. The scale of that freely-available data is what made it work.

The objective is precise. For a batch of N pairs, embed the images and captions, L2-normalize, and form the N×N cosine-similarity matrix scaled by a learned temperature. The correct match for row i is column i — the diagonal — so it's just cross-entropy, run both directions:

One batch of N pairs supplies N positives and N²−N negatives for free — which is why a huge batch is part of the recipe.

Pull the true pair together, push all N−1 impostors in the row apart, symmetrically for columns. That's the entire training signal — and the runnable version below builds the matrix, computes the loss, and does zero-shot classification with it.

Here's the trick that stunned people. Because CLIP compares images to text, you can classify an image into categories it was never trained on — just describe them. Want to tell cats from dogs? Embed the phrases "a photo of a cat" and "a photo of a dog," embed the image, and pick whichever text is closest.

CLIP became the connective tissue of multimodal AI. Text-to-image models lean on it to steer generation — measuring how well an emerging image matches the prompt, so a diffusion model can be nudged toward your words. It powers multimodal search ("find photos that look like this sentence"), content moderation, and image captioning, and its contrastive recipe seeded a wave of vision-language models.

Its lasting idea is simple and profound: put different kinds of data into one shared space and comparison becomes trivial. Once images and words share a map, the boundary between seeing and reading starts to dissolve.

CLIP is two encoders, one similarity matrix and one loss, trained from scratch on 400 million image–text pairs from the web. The paper's real work is in the details: how the data was gathered, how a batch of 32,768 pairs is scored, and how the trained model becomes an ImageNet classifier without seeing a single ImageNet label.

Training

1. Collect the pairs. The dataset, called WIT (WebImageText), was built by searching for image–text pairs whose text contains one of 500,000 queries: every word that appears at least 100 times in English Wikipedia, plus bi-grams with high pointwise mutual information, Wikipedia article titles above a search-volume threshold and WordNet synsets. At most 20,000 pairs were kept per query, so common concepts could not swamp rare ones.
2. Encode each image. Eight image encoders were trained: five ResNets (ResNet-50, ResNet-101 and three scaled-up versions using about 4, 16 and 64 times the compute of ResNet-50), with an attention-pooling layer replacing global average pooling, and three Vision Transformers (ViT-B/32, ViT-B/16, ViT-L/14). The only data augmentation is a random square crop of the resized image.
3. Encode each caption. A 63-million-parameter Transformer with 12 layers, width 512 and 8 attention heads reads lower-cased byte-pair-encoded text (a 49,152-token vocabulary, at most 76 tokens). Its top-layer activation at the end-of-text token, layer-normalized, is the text feature. Self-attention is masked left to right so that a pretrained language model could be dropped in later.
4. Project and normalize. One linear projection per encoder, with no nonlinear head, maps into the shared space, and every vector is scaled to unit length, so a dot product is a cosine.
5. Score the batch. Every image is compared with every caption in a batch of 32,768, the scores are multiplied by a learned scale that starts at 1/0.07 and is capped at 100, and the symmetric cross-entropy in the next section is applied. Each GPU computes only the similarities its own slice of the batch needs.
6. Run. Every model trains for 32 epochs with Adam and decoupled weight decay, a cosine learning-rate schedule, mixed precision and gradient checkpointing. The largest ResNet took 18 days on 592 V100 GPUs and the largest Vision Transformer 12 days on 256. The best model, ViT-L/14@336px, adds one epoch at 336-pixel resolution.

From a trained model to a classifier

The paper treats the text encoder as a generator of classifier weights. Each class name is written into a prompt, encoded and normalized; the resulting vectors, one per class, are the weight matrix of a linear classifier over image embeddings, and the predicted class is the one with the largest scaled dot product. Nothing is trained, so changing the label set means editing a list.

Two refinements matter. Prompt engineering: the template “A photo of a {label}.” beat the bare class name by 1.3 points on ImageNet, and dataset-specific context helped further, such as “a type of pet” for Oxford Pets or “a satellite photo of” for remote sensing. Ensembling: for ImageNet the authors averaged the normalized embeddings of 80 templates, such as “a photo of a big …” and “a photo of a small …”, which added another 3.5 points. Averaging in embedding space leaves one vector per class, so the ensemble costs nothing extra per image.

The second evaluation is a linear probe: a logistic regression trained on frozen image features with each dataset's own labels, the standard test of representation quality.

• ui, vj — unit-length image and text embeddings, so ⟨u_i, v_j⟩ is a cosine between −1 and 1
• first term — each image picks its own caption out of N; second term — each caption picks its own image
• s — the logit scale; the network learns t = log s, which keeps s positive
• wc — the zero-shot weight vector for class c, averaged over K prompt templates

At the extremes: if s were near 0, every logit would be near 0, every row of the softmax would be uniform, and the loss would sit at ln N whatever the embeddings, about 10.4 nats at N = 32,768, with almost no gradient. As s grows, small cosine differences become large logit differences: at s = 100 a cosine gap of 0.05 is a logit gap of 5, a factor of e^⁵ ≈ 148 in probability, so the softmax concentrates on the hardest negatives. The cap at 100 exists because the paper found that letting the scale grow further made training unstable.

The eight models were evaluated two ways: zero-shot, with classifiers written from class names, on more than 30 datasets, and with linear probes on 27. Robustness was measured on natural distribution shifts of ImageNet.

The last two rows use the same frozen image features. Fitting them to ImageNet's labels adds 9.2 points on ImageNet and helps on ImageNetV2, a re-collection of ImageNet, but loses on the other four shifted sets, so the robustness comes from defining classes in language rather than from the features alone. Overall, zero-shot CLIP closes up to 75% of the gap between ImageNet accuracy and accuracy under distribution shift.

Two ablations explain the design. A Transformer trained to predict each caption word for word learned ImageNet classes three times more slowly than a model predicting the caption's bag of words, and switching that to the contrastive objective gave a further fourfold gain. Across 36 datasets, prompt engineering and ensembling added almost 5 points on average, similar to the gain from 4 times more compute.

The failures are reported too: 88% on handwritten MNIST digits, below a logistic regression on raw pixels; weak results on satellite images, tumor detection, counting and traffic signs; and an estimated thousandfold increase in compute before zero-shot CLIP would reach the overall state of the art.

The paper is unusually thorough about its own weaknesses. Performance is poor on fine-grained distinctions, on counting, on abstract or systematic tasks, and on anything far from what web captions describe — the model learned the distribution of things people photograph and write about, which is not the distribution of things one might want recognised. Results are also sensitive to prompt wording, so a reported zero-shot number embeds prompt engineering, and the paper says so.

"Zero-shot" deserves the scrutiny it later received: with 400 million uncurated pairs, whether a category was genuinely unseen is unanswerable, so the term means "no task-specific training" rather than "no exposure". And because the supervision is uncurated web text, the space encodes the associations of that text, which the paper examines directly rather than leaving to critics.

How it held up: as the substrate for nearly everything multimodal. A shared image-text space is the text conditioning in latent diffusion, the backbone of cross-modal retrieval, and — most consequentially — the visual encoder that rungs 2 and 3 attach to a language model. This rung aligned the two modalities; the rest of the ladder is about what you can then build on top of that alignment.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Joining text to images, audio, video and finally to a robot arm. Starts with the contrastive objective the whole field standardised on.

1. 1CLIP2021
2. 2Flamingo2022
3. 3LLaVA2023
4. 4wav2vec 2.02020
5. 5Whisper2022
6. 6V-JEPA2024
7. 7RT-22023
8. 8Genie2024

All 110 papers, in 19 ladders →