CLIP,
explained.

Rung 1, so the comparison is against supervised image classification. A classifier is trained against a fixed list of categories, and that list is the ceiling on what it can ever say — recognising a new kind of thing means new labelled data and a new output layer. The labels are also expensive, which caps the data.

The change is to supervise with language instead of labels. Train an image encoder and a text encoder together on image-caption pairs so that a picture and its own caption land close together in one shared space, and mismatched pairs land apart — the contrastive objective from the vision ladder, with a caption in place of a second crop. The supervision now comes from text that already existed alongside the images, so the training set is roughly 400 million pairs scraped from the web rather than a hand-annotated corpus.

What falls out is the property that made it famous. Because the "labels" are arbitrary sentences, classification becomes a retrieval: write out candidate class names as short phrases, encode them, and pick whichever is nearest the image. You can classify into categories that were never a training target, define new ones at inference time by typing them, and — stated in units — match a strong supervised model on ImageNet without training on ImageNet at all. The fixed output layer is gone, replaced by a text encoder.

Before CLIP, a vision model was trained on a fixed list of labels — say, the 1,000 categories of ImageNet — and could only ever recognize those. Ask it about anything outside its list and it was helpless. Worse, building it meant an army of humans hand-labeling millions of images into rigid buckets.

Meanwhile language models understood rich, open-ended text. The two never met. What if a model could learn about images the way a person browsing the web does — from pictures paired with the captions people wrote for them?

CLIP — Contrastive Language-Image Pre-training — trains two encoders at once: one that turns an image into a vector, one that turns text into a vector. The goal is to align them so that a picture and its matching description land at nearly the same spot in a single shared embedding space.

Once images and words live on the same map, you can measure how well any picture matches any sentence just by checking how close their vectors are. Pixels and language become directly comparable.

Take a batch of image-caption pairs. Embed every image and every caption. Now train so each image is closest to its own caption and far from every other caption in the batch — pull the true pairs together, push the impostors apart.

Done across 400 million image-caption pairs scraped from the web — natural-language supervision, no hand-labeling — the two encoders converge into one aligned space. The scale of that freely-available data is what made it work.

The objective is precise. For a batch of N pairs, embed the images and captions, L2-normalize, and form the N×N cosine-similarity matrix scaled by a learned temperature. The correct match for row i is column i — the diagonal — so it's just cross-entropy, run both directions:

One batch of N pairs supplies N positives and N²−N negatives for free — which is why a huge batch is part of the recipe.

Pull the true pair together, push all N−1 impostors in the row apart, symmetrically for columns. That's the entire training signal — and the runnable version below builds the matrix, computes the loss, and does zero-shot classification with it.

Here's the trick that stunned people. Because CLIP compares images to text, you can classify an image into categories it was never trained on — just describe them. Want to tell cats from dogs? Embed the phrases "a photo of a cat" and "a photo of a dog," embed the image, and pick whichever text is closest.

CLIP was remarkably robust and general: it held up across dozens of datasets and unusual image styles where narrow supervised models fell apart, because it learned from the messy diversity of the whole internet rather than one curated dataset.

CLIP became the connective tissue of multimodal AI. Text-to-image models lean on it to steer generation — measuring how well an emerging image matches the prompt, so a diffusion model can be nudged toward your words. It powers multimodal search ("find photos that look like this sentence"), content moderation, and image captioning, and its contrastive recipe seeded a wave of vision-language models.

Its lasting idea is simple and profound: put different kinds of data into one shared space and comparison becomes trivial. Once images and words share a map, the boundary between seeing and reading starts to dissolve.

The paper is unusually thorough about its own weaknesses. Performance is poor on fine-grained distinctions, on counting, on abstract or systematic tasks, and on anything far from what web captions describe — the model learned the distribution of things people photograph and write about, which is not the distribution of things one might want recognised. Results are also sensitive to prompt wording, so a reported zero-shot number embeds prompt engineering, and the paper says so.

"Zero-shot" deserves the scrutiny it later received: with 400 million uncurated pairs, whether a category was genuinely unseen is unanswerable, so the term means "no task-specific training" rather than "no exposure". And because the supervision is uncurated web text, the space encodes the associations of that text, which the paper examines directly rather than leaving to critics.

How it held up: as the substrate for nearly everything multimodal. A shared image-text space is the text conditioning in latent diffusion, the backbone of cross-modal retrieval, and — most consequentially — the visual encoder that rungs 2 and 3 attach to a language model. This rung aligned the two modalities; the rest of the ladder is about what you can then build on top of that alignment.

CLIP's core idea is a shared embedding space for images and text: train an image encoder and a text encoder together so a picture of a dog and the words "a photo of a dog" land near each other. Once images and captions live in the same space, you can compare across modalities with a dot product — the foundation of multimodal AI.

The training trick was contrastive learning at scale: given a batch of image–text pairs, pull each real pair together and push all the mismatched pairs apart. No hand-labeled categories — just 400M noisy image–caption pairs from the web. That weak, abundant supervision is what let it learn broad visual concepts instead of a fixed label set.

The headline result was zero-shot classification: to classify an image, embed candidate labels as text ("a photo of a cat", "a photo of a car") and pick the nearest. No task-specific training or fine-tuning — a single model classifies categories it was never explicitly trained on, just by comparing embeddings. That flexibility was a genuine break from the train-one-classifier-per-task era.

CLIP became infrastructure. Its text–image space guides text-to-image generators (steering diffusion toward a prompt), powers image search and retrieval, filters and captions training data, and feeds the vision side of multimodal LLMs. The lasting lesson is the one that keeps recurring in AI: learn a good shared representation from cheap, plentiful data, and a huge range of tasks become "just" nearest-neighbor lookups in that space.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

Joining text to images, audio, video and finally to a robot arm. Starts with the contrastive objective the whole field standardised on.

1. 1CLIP2021
2. 2Flamingo2022
3. 3LLaVA2023
4. 4wav2vec 2.02020
5. 5Whisper2022
6. 6V-JEPA2024
7. 7RT-22023
8. 8Genie2024

All 110 papers, in 19 ladders →