SimCLR,
explained.

Rungs 1 and 2 both need a labelled image for every training signal, and rung 2 needs an enormous number of them. That is the binding constraint on vision, because labels are bought and pixels are free. SimCLR's change is to construct the training signal out of the images alone.

The construction is comparison. Take one image, augment it twice, and require the two views to land close together in representation space, while every other image in the batch is pushed away. Nothing in that objective needs to know what the image contains — only that these two crops came from the same photograph and those did not. The loss (NT-Xent) is a softmax over similarities with a temperature that sets how sharply near-misses are punished.

Stated in the ladder's units: rung 2 buys quality with labelled data; SimCLR buys it with batch size and augmentation design. Because the negatives are just the other images in the batch, a bigger batch means more negatives and a harder, more informative task — the paper works at batches in the thousands, which is a hardware requirement rather than a hyperparameter. And the augmentation set is not decoration: the paper shows that random cropping alone is solvable by a shortcut, because two crops of one photo share a colour histogram, so colour distortion has to be composed with it to force the model to use shape and content. The composition, not any single augmentation, is the method.

Labels are expensive; raw images are nearly free. Self-supervised learning tries to build good representations from unlabeled data by inventing a task the data itself can grade. SimCLR's invented task is contrastive: given a representation, judge whether two inputs are "the same thing seen differently" or "different things." No categories, no annotations — just same-or-different.

The elegant part is where the supervision comes from. Take one image, apply two random transformations — crop it here, jitter its colors, blur it — and you have two views that you know depict the same content. That known relationship is the label the model never had. Learn to pull those two views together in representation space while pushing away every other image, and you have learned what makes an image what it is, invariant to the incidental stuff.

Concretely: take a batch of images. For each one, create two augmented views. The two views of the same image are a positive pair — they should end up close. Every view of a different image in the batch is a negative — it should end up far. So each anchor view has exactly one positive to attract and many negatives to repel, all supplied by the batch itself for free.

Nothing here required a human to say "this is a dog." The batch is its own answer key: same image, positive; different image, negative. That is what makes it self-supervised, and why more images in a batch simply means more negatives to learn from.

The loss that enforces all this is NT-Xent — normalized, temperature-scaled cross-entropy. For an anchor view i with positive partner j, it is a softmax classification where the "correct class" is the positive and every negative is a wrong class. Similarities are cosine (that's the "normalized"), scaled by a temperature τ:

Minimizing this maximizes the positive's share of the softmax — it rises as the positive similarity grows, and falls as any negative creeps closer, so one loss both attracts the positive and repels negatives. The temperature τ controls how sharply the model focuses on the hardest (closest) negatives — smaller τ, sharper focus. And because the denominator sums over every negative, a bigger batch makes the task harder and the features stronger. The runnable version below shows the loss dropping as the positive dominates and rising as negatives multiply.

SimCLR's other contribution was measuring what actually matters. Three choices proved decisive:

The projection-head finding is a subtle, influential one: the representation you train the contrastive loss on is not the best representation to use downstream. Keeping the pre-projection features — which retain more information the loss would have thrown away — is now standard practice.

SimCLR has four parts: a stochastic augmentation, an encoder, a small projection head and a contrastive loss. The default setup trains a ResNet-50 for 100 epochs at batch size 4,096 for ablations, and much longer for the final results.

1. Sample a batch. Draw N images, 4,096 by default. No labels are read.
2. Make two views of each image. Two augmentations are drawn from the same family and applied independently: a random crop (with flip) resized back to full size, then color distortion (brightness, contrast, saturation, hue and random grayscale), then Gaussian blur. The batch now holds 2N views, and the two views of one image form a positive pair.
3. Encode. A ResNet-50 turns each view into a 2,048-dimensional vector h, taken after average pooling. Both views share the same weights.
4. Project. A two-layer MLP with one ReLU hidden layer maps h to a 128-dimensional vector z. The loss sees z, never h.
5. Compare everything with everything. The z vectors are L2-normalized and every pair among the 2N views gets a cosine similarity. For a given view, its partner is the one positive and the other 2N − 2 views, including both views of every other image, are negatives. With N = 4,096 that is 8,190 negatives per anchor, all supplied by the batch for free.
6. Apply NT-Xent. Each anchor's loss is a softmax cross-entropy over the other 2N − 1 views, scaled by a temperature, with the positive as the correct class. It is averaged over all 2N anchors, taking each view in turn as the anchor.
7. Keep batch statistics honest. Batch-norm means and variances are aggregated over all devices. With per-device statistics, the two views of an image sit on the same device and can leak information about each other through the statistics, letting the network lower the loss without learning better features.
8. Optimize with LARS. Large batches destabilize plain SGD with momentum, so SimCLR uses the LARS optimizer at every batch size, with learning rate 0.3 × batch size / 256 (4.8 at 4,096), weight decay of 10^⁻⁶, ten epochs of linear warmup and cosine decay. One 100-epoch run at batch size 4,096 took about 1.5 hours on 128 TPU v3 cores.
9. Throw the head away. After training, g is discarded and h is the representation. To evaluate it, freeze the encoder and train a linear classifier on h (linear evaluation), or fine-tune the whole encoder on a small labeled fraction.

What the ablations established

The augmentation grid applied transformations alone and in pairs. No single one was enough, and the pair that stood out was random cropping with color distortion. The paper's explanation is a shortcut: crops of one photo share a color histogram, and a network can match them by color alone, so color distortion has to be composed with cropping to remove that cue. Stronger color distortion helped contrastive training steadily while slightly hurting a supervised network, which means self-supervision needs more aggressive augmentation than supervision does.

The projection head is the second finding. A nonlinear head beat a linear one by about three points and no head by more than ten, and the layer before the head carried a representation more than ten points better than the layer after it.

• i, j — the two views of one image; k runs over every view in the batch except i itself, so the sum has 2N − 1 terms, one of them the positive.
• τ — the temperature. A small τ makes the softmax concentrate on the most similar negatives; a large τ weights all negatives nearly equally.
• g, σ — the two-layer projection head with a ReLU; L averages over both orderings of each pair.

To see the temperature at work, take two negatives with similarities 0.9 and 0.5 to the anchor. At τ = 0.1 the harder one gets exp(4) ≈ 55 times the weight of the easier one; at τ = 1 only exp(0.4) ≈ 1.5 times. In the paper's sweep with normalized embeddings, τ = 0.1 gave the best linear accuracy, and dropping the normalization made the contrastive task easier but the representation worse.

The ablations use ResNet-50 for 100 epochs on ImageNet and report top-1 accuracy of a linear classifier on frozen features. The headline results train 1,000 epochs with wider ResNet-50 variants (2× and 4× width).

The 76.5 matches a supervised ResNet-50. With few labels, fine-tuned SimCLR reached these top-5 accuracies:

Ablations

• Loss. NT-Xent reached 63.9, against 57.9 for a logistic loss and 57.5 for a margin loss even with semi-hard negative mining.
• Batch and length. At 100 epochs, batch 256 gave 57.5 and batch 4,096 gave 64.6; at 800 epochs the gap shrank to 66.6 versus 68.9.
• Color strength. From 1/8 to full strength, accuracy rose from 59.6 to 63.2, and to 64.5 with blur, while a supervised ResNet fell from 77.0 to 75.4.
• Information in h. A small classifier predicting whether an image was rotated reached 67.6 percent from h but only 25.6 from z, where chance is 25.

The costs are reported without softening: large batches, long training, and accelerator counts that put the method out of reach for most groups at the time. The paper also documents a result it does not explain — the representation used for the contrastive loss is worse for downstream tasks than the layer just before it, which is why a projection head is trained and then thrown away. A component whose purpose is to be discarded is a strong hint that the objective and the goal are not the same thing, and that gap is what rungs 4 and 5 attack from two directions.

The batch-size dependence was the part the field refused to accept, and it was attacked successfully from both sides. MoCo, which actually predates SimCLR, had already kept a queue of representations from previous batches so the negatives outlive the batch that produced them; BYOL removed negatives altogether and still avoided collapse. Between them they established that the large batch was a property of this implementation, not of contrastive learning.

How it held up: the specific recipe is largely of historical interest for pure vision, where rungs 4 and 5 are the defaults. The objective is one of the most consequential ideas on this site — pull matched pairs together, push everything else apart — because it is exactly what CLIP does with image-text pairs instead of two crops, and what the embedding models on the retrieval ladder are trained with. Read this rung as the loss function's paper, not the vision model's.

SimCLR crystallized the contrastive family of self-supervised learning — learn by comparison, pull positives together and push negatives apart — and its influence runs straight into multimodal work: CLIP uses the same contrastive idea across images and text. The lineage of related methods (the earlier MoCo, then BYOL and SimSiam) mostly explored how to keep the quality while dropping the need for huge batches of negatives.

It's illuminating to set it beside masked autoencoders, the other great self-supervised family. Contrastive methods need augmentations and negatives and learn invariances; masked methods need neither and learn by reconstruction. Two very different routes to the same destination — strong features without labels — and modern vision draws on both. SimCLR is the clearest statement of the contrastive route.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

From the network that started the deep-learning era to one you can segment anything with — and the shift from labels to self-supervision in between.

1. 1AlexNet2012
2. 2Vision Transformer2021
3. 3SimCLR2020
4. 4Masked Autoencoders2022
5. 5DINOv22023
6. 6Segment Anything2023

All 110 papers, in 19 ladders →