CLIP,
explained.

Before CLIP, a vision model was trained on a fixed list of labels — say, the 1,000 categories of ImageNet — and could only ever recognize those. Ask it about anything outside its list and it was helpless. Worse, building it meant an army of humans hand-labeling millions of images into rigid buckets.

Meanwhile language models understood rich, open-ended text. The two never met. What if a model could learn about images the way a person browsing the web does — from pictures paired with the captions people wrote for them?

CLIP — Contrastive Language-Image Pre-training — trains two encoders at once: one that turns an image into a vector, one that turns text into a vector. The goal is to align them so that a picture and its matching description land at nearly the same spot in a single shared embedding space.

Once images and words live on the same map, you can measure how well any picture matches any sentence just by checking how close their vectors are. Pixels and language become directly comparable.

Take a batch of image-caption pairs. Embed every image and every caption. Now train so each image is closest to its own caption and far from every other caption in the batch — pull the true pairs together, push the impostors apart.

Done across 400 million image-caption pairs scraped from the web — natural-language supervision, no hand-labeling — the two encoders converge into one aligned space. The scale of that freely-available data is what made it work.

The objective is precise. For a batch of N pairs, embed the images and captions, L2-normalize, and form the N×N cosine-similarity matrix scaled by a learned temperature. The correct match for row i is column i — the diagonal — so it's just cross-entropy, run both directions:

One batch of N pairs supplies N positives and N²−N negatives for free — which is why a huge batch is part of the recipe.

Pull the true pair together, push all N−1 impostors in the row apart, symmetrically for columns. That's the entire training signal — and the runnable version below builds the matrix, computes the loss, and does zero-shot classification with it.

Here's the trick that stunned people. Because CLIP compares images to text, you can classify an image into categories it was never trained on — just describe them. Want to tell cats from dogs? Embed the phrases "a photo of a cat" and "a photo of a dog," embed the image, and pick whichever text is closest.

CLIP was remarkably robust and general: it held up across dozens of datasets and unusual image styles where narrow supervised models fell apart, because it learned from the messy diversity of the whole internet rather than one curated dataset.

CLIP became the connective tissue of multimodal AI. Text-to-image models lean on it to steer generation — measuring how well an emerging image matches the prompt, so a diffusion model can be nudged toward your words. It powers multimodal search ("find photos that look like this sentence"), content moderation, and image captioning, and its contrastive recipe seeded a wave of vision-language models.

Its lasting idea is simple and profound: put different kinds of data into one shared space and comparison becomes trivial. Once images and words share a map, the boundary between seeing and reading starts to dissolve.