DINOv2,
explained.

The two rungs below get their signal from outside the network — SimCLR from other images, MAE from pixels it hid. This rung gets it from a copy of itself. A student network is trained to match the output of a teacher, and the teacher is an exponential moving average of the student. The student sees a local crop, the teacher a global one, so the task is to predict the whole from the part, in representation space rather than pixel space.

Stated plainly, that objective should collapse. Nothing pushes representations apart — there are no negatives — so mapping every image to one constant vector satisfies it perfectly. What prevents it is a pair of opposed forces applied to the teacher's output: centering, which subtracts a running mean and so stops any single dimension from dominating, and sharpening via a low temperature, which stops the output from flattening into the uniform distribution that centering alone would encourage. Collapse is avoided by balancing two ways of collapsing against each other.

The delta in the units rung 4 left open: these features are strong frozen. Under a linear probe or plain k-nearest-neighbour classification with no fine-tuning at all, self-distilled features beat masked-reconstruction features clearly — which is precisely MAE's admitted weakness. DINOv2 then scales the recipe, and its headline claim is about generality: one frozen backbone serving classification, retrieval, depth estimation and segmentation without task-specific training.

Self-supervised vision has three great families, each learning strong features without labels but by a different mechanism. Contrastive methods pull augmented views of one image together and push other images apart — they need negatives. Masked methods hide parts of an image and reconstruct them — they need a reconstruction target. Self-distillation — DINO's family — needs neither.

Its idea: don't compare against other images or reconstruct pixels. Just have one network (the student) learn to match the output of another (the teacher) on different views of the same image. The twist that makes it self-supervised is that the teacher isn't a separate, pretrained model — it's a slowly-updated copy of the student. There is no external signal at all; the model teaches itself. That should be a recipe for collapse, and the interesting engineering is why it isn't.

Both networks have the same architecture. The student is trained by gradient descent to make its output distribution match the teacher's on the same image (different augmentations — typically the student sees smaller local crops, the teacher global ones, so the student learns "local-to-global" correspondence). The teacher is not trained by gradients. Its weights are an exponential moving average (EMA) of the student's:

With a momentum m near 1, the teacher barely moves each step — it's a stable, lagged version of the student. As the student improves, the teacher inherits those improvements a beat later, and the student then chases the slightly-better teacher. The two bootstrap upward together. The runnable version below shows the EMA barely nudging the teacher per step, which is exactly what gives training its stability.

The obvious failure mode: student and teacher both learn to output the same constant for every image. That trivially minimizes the matching loss and learns nothing — collapse. DINO prevents it with two opposing operations applied to the teacher's output, which balance each other:

Centering alone would flatten everything toward uniform; sharpening alone would let one dimension run away. Applied together, they cancel each other's failure modes and leave a stable regime where the teacher's targets are neither degenerate constant nor uniform noise — exactly the informative signal the student needs. That delicate balance is the real trick that makes teacherless self-distillation work.

DINO (2021) established the method on ViTs and made a striking observation: its attention maps segmented objects for free, without ever being told what an object is. DINOv2 (2023) scaled the recipe into a foundation model for vision. Its contributions were less about the core loss and more about making it work at scale:

The result was a set of visual features you can pull off the shelf: freeze DINOv2, put a light head on top, and get strong results on classification, segmentation, depth estimation, and retrieval. It became a go-to image backbone precisely because those features generalized so broadly.

DINOv2 is DINO's student and teacher idea with three changes: an extra patch-level loss borrowed from iBOT, a different way of keeping the teacher's outputs balanced, and a regularizer that spreads features apart. Most of the gain comes from assembling the training data and from engineering that made the combination train stably at scale.

1. Collect and deduplicate. Images come from a crawl of web pages, filtered for unsafe content and with faces blurred, giving on the order of a billion unique images. A copy-detection pass removes near-duplicates, and a stricter pass removes anything too similar to the train or test sets of the evaluation benchmarks.
2. Retrieve what looks like curated data. A self-supervised ViT-H/16 trained on ImageNet-22k embeds every image. Using cosine similarity, the pipeline pulls from the crawl the images closest to a set of curated sources: ImageNet-22k, the ImageNet-1k training set, Google Landmarks and several fine-grained datasets. For large sources it takes about four neighbors per query image; for small ones it samples from the matching cluster. The result is LVD-142M, built with no captions or metadata, in under two days on 20 nodes of 8 GPUs.
3. Cut views. Each image gives two global crops at 224 pixels and several local crops at 98 pixels. With 14×14 patches those are 256 and 49 tokens. The student sees all crops; the teacher sees only the global ones.
4. Image-level loss. A student and a teacher ViT with the same architecture each pass their class token through a projection head to scores over a large set of prototypes. The student is trained with cross-entropy to match the teacher's distribution. The teacher is an exponential moving average of the student, with momentum rising from 0.994 to 1 on a cosine schedule.
5. Patch-level loss. Some patches of the student's input are masked. The student's second head must predict, at each masked position, the distribution the teacher's matching head produces for the same visible patch. This is the iBOT objective, and it is what makes the features good at dense tasks. Unlike iBOT, the two heads do not share weights; at scale the paper found the opposite of the smaller-scale result.
6. Balance the teacher. Instead of subtracting a running mean from the teacher's output, DINOv2 applies Sinkhorn-Knopp normalization for three iterations, which makes the batch's assignments to prototypes roughly even.
7. Spread the features. A KoLeo term, weighted 0.1, takes the class tokens of the first global crop within each GPU's batch, normalizes them and penalizes any point whose nearest neighbor is too close.
8. Finish at high resolution. For about 10,000 iterations at the end, images are raised to 518×518 pixels, which helps tasks with small objects while avoiding the cost of training at that size throughout.
9. Distill the smaller models. The 1.1-billion-parameter ViT-g/14 is trained from scratch. ViT-S, B and L are trained by distillation from it using the same loop, with the large model as a frozen teacher, no masking and no stochastic depth, and an average of the student weights as the released model.

The engineering that made it possible

The authors wrote their own memory-efficient attention, packed the 224 and 98 pixel sequences into one long sequence with a block-diagonal mask, skipped the computation of dropped residuals when applying stochastic depth at 40 percent, and sharded the model with fully sharded data parallelism. Together these made the code about twice as fast as the iBOT implementation at a third of the memory.

• ps, pt — student and teacher distributions over prototypes; the teacher's is balanced with Sinkhorn-Knopp normalization. The first loss uses class tokens, the second masked patch tokens.
• xi — an L2-normalized class token; di is the distance from xi to its nearest neighbor in the batch.
• m — the teacher's momentum, following a cosine schedule.

Take four normalized features whose nearest-neighbor distances are 0.5, 0.5, 1 and 1: the KoLeo term is −(ln 0.5 + ln 0.5 + 0 + 0)/4 ≈ 0.35. If one pair collapses to a distance of 0.01, it becomes about 2.3, so near-duplicate features are punished hard.

All headline numbers use frozen features: a linear classifier or nearest-neighbor lookup, with no backbone fine-tuning. ImageNet-1k linear accuracy, with k-NN in brackets:

On ADE20k segmentation with a linear head, ViT-g/14 scored 49.0 mIoU against 39.3 for OpenCLIP ViT-G/14 and 33.3 for MAE. Fine-tuning the backbone added about 2 points on ImageNet, so it is optional.

Ablations

• Recipe. Eleven changes took an iBOT baseline on ImageNet-22k from 72.9 to 82.0 k-NN, with no new data.
• KoLeo. Retrieval rose from 55.6 to 63.9 and ImageNet linear from 85.3 to 85.8.
• Patch loss. Removing it cost about 3 points on ADE20k.
• Sinkhorn-Knopp. Swapping it in changed neither ImageNet number.
• Distillation. A ViT-L trained from scratch scored 84.5; distilled from ViT-g it scored 86.3.

The anti-collapse mechanism is acknowledged as a balance rather than a guarantee — the two forces have to be tuned against each other, and the papers discuss the failure modes on either side of that balance. This is the clearest case on the ladder of a method that works reliably in practice without a principled reason that it must.

DINOv2's own gains are attributed in part to data curation — an automated pipeline that assembles and deduplicates a large uncurated pool into a balanced training set — though the paper also credits training-recipe changes (an iBOT baseline on the same data rose from 72.9 to 82.0 k-NN accuracy on recipe changes alone). The curation is stated openly, and it is the part hardest for anyone else to reproduce, because the pipeline and the resulting mixture are the product. A result whose main ingredient is a dataset you cannot obtain is a strong result and a weak experiment, and both of those are worth saying.

How it held up: better than almost anything else on this ladder in terms of day-to-day use. Frozen DINOv2 features are a default off-the-shelf visual representation, and the discovery that attention maps from this family segment objects without ever being trained to is one of the more striking unprompted behaviours in vision. The honest limitation is the one the next rung makes vivid: these are excellent features with no interface. Nothing about them lets a user point at something and get an answer, and that gap is a product problem rather than a representation problem.

DINO and DINOv2 cemented self-distillation as a first-class self-supervised approach and gave vision a genuinely useful foundation backbone trained without labels. The finding that its features segment objects on their own hinted at something deep — that the right unsupervised objective recovers structure we usually think requires supervision. DINOv2 turned that hint into a practical, widely-used model.

Set beside its siblings, it completes a picture of self-supervised vision: contrastive (compare, with negatives), masked (reconstruct), and self-distillation (match a momentum teacher). Three different bets on what makes a good unsupervised objective, all reaching strong features — and DINOv2 is the clearest demonstration that the self-distillation bet, scaled with curated data, produces backbone-quality features. The through-line across all three: the label was never the only teacher.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

From the network that started the deep-learning era to one you can segment anything with — and the shift from labels to self-supervision in between.

1. 1AlexNet2012
2. 2Vision Transformer2021
3. 3SimCLR2020
4. 4Masked Autoencoders2022
5. 5DINOv22023
6. 6Segment Anything2023

All 110 papers, in 19 ladders →