Segment Anything,
explained.

Rungs 1 to 5 produce features, and something else has to be trained on top to do a task. SAM's change is to make the task itself promptable and general: given a point, a box or a rough mask, return the mask of the object that was indicated, zero-shot, for objects and domains never seen in training. The model is not a segmenter for a fixed label set; it is a function from a pointing gesture to a cut-out.

Two design choices follow from wanting that to be usable. The first is a split by cost: a heavy transformer image encoder runs once per image, and a very light prompt encoder and mask decoder run per click, so after the one-off encode each subsequent prompt is answered fast enough to feel interactive. The second is explicit handling of ambiguity — a click on a person's shirt might mean the shirt, the person, or the group — so the model emits several masks with confidence scores rather than averaging incompatible interpretations into one bad mask.

The units that actually explain the result are not architectural. SAM is trained on roughly a billion masks over about 11 million images, and that dataset did not exist; it was built by a data engine that ran in stages, with annotators correcting the model, then the model proposing and annotators only handling what it missed, then the model annotating automatically at scale. Each stage's model was trained on the previous stage's output. The model and the dataset bootstrapped each other, and that loop is the transferable idea on this rung.

Segmentation is pixel-level understanding: not just "there's a dog," but exactly which pixels are the dog, traced to its outline. It powers photo editing, medical imaging, robotics, self-driving — anywhere you need to isolate an object precisely rather than just draw a loose box.

The old workflow was narrow and expensive. You picked a fixed set of categories, collected a dataset with every object masked by hand (painstaking, pixel-perfect labeling), and trained a specialist model. It worked only for those categories, on that kind of image. A new domain meant a new dataset and a new model. Segmentation had never had its "foundation model" moment — a single pre-trained model you could point at anything.

SAM's reframing borrows from language models: make segmentation promptable. Instead of "segment these 80 fixed classes," the task becomes "given a prompt pointing at something, return its mask." The prompt is deliberately flexible — a single click on the object, a bounding box around it, a rough mask, even (in principle) text.

This one shift generalizes segmentation the way prompting generalized language. The model no longer needs a predefined vocabulary of objects; it responds to whatever you indicate. Train it to be good at "return the mask for this prompt" across enough examples, and it can segment novel objects it never saw a category label for — the definition of a foundation model for this task.

SAM is engineered around one practical goal: respond to prompts interactively, in real time. It splits into three pieces so the expensive work happens only once per image.

The image encoder — a heavy Vision Transformer — processes the whole image into an embedding a single time. The lightweight prompt encoder turns a click or box into a small vector. The fast mask decoder then fuses the image embedding with the prompt to produce a mask in milliseconds. So after a one-time encode, you can click around an image and get instant masks — the responsiveness that made SAM feel magical.

A single click is genuinely ambiguous. Click on someone's shirt — do you mean the shirt, the person wearing it, or the whole group they're standing in? There's no single correct answer, and a model that always guesses one will often guess wrong.

SAM's fix is to embrace it: for an ambiguous prompt, it outputs several valid masks at once — the shirt, the person, the group — each with a confidence score. The user (or a downstream system) picks the intended one. This multi-mask output turns ambiguity from a failure mode into a feature, and it's a big part of why SAM feels so usable in practice.

Mask quality is measured by IoU — the overlap between prediction and ground truth over their union — and that's what makes the multi-mask trick pay off:

A model forced to one answer is wrong whenever you meant a part; proposing shirt/person/group and ranking them means the right mask is always on offer. The runnable version below computes the IoU table and the pick.

A foundation model needs a foundation-scale dataset, and no billion-mask segmentation dataset existed — masks are far more expensive to label than image tags. So Meta built a data engine: a bootstrapping loop where model and data improve each other.

It ran in three gears. Assisted: annotators labeled masks with an early SAM helping, much faster than from scratch. Semi-automatic: SAM proposed masks for confident objects and humans filled the gaps. Fully automatic: SAM, now strong, blanketed each image with a grid of point prompts and generated masks on its own. Each round of new labels retrained a better SAM, which labeled more, faster. The output was SA-1B — over 1 billion masks across 11 million images, the largest segmentation dataset ever, most of it machine-generated but human-validated.

Trained on SA-1B, SAM segments objects zero-shot — cutting out things it was never explicitly trained to recognize, across wildly different image types, from a simple click. It matched or beat many specialist, fully-supervised segmentation models on datasets it had never seen, the hallmark of a true foundation model.

Meta released the model, the code, and the entire SA-1B dataset. Because SAM produces reusable masks from cheap prompts, it slotted in as a building block everywhere: image and video editing, medical and satellite imaging, robotics, and as the segmentation front-end for larger multimodal systems. It became infrastructure almost immediately.

SAM cuts out masks but doesn't name them — it tells you where an object is, not what it is. Semantic labels need a separate model. Its heavy ViT image encoder is compute-hungry, making the original SAM impractical on-device or for real-time video (later work like MobileSAM and SAM 2 tackled speed and video). And on very fine or thin structures, its masks can miss detail.

Text prompting, teased in the paper, was more proof-of-concept than robust feature. And like any model, SA-1B's automatically-generated labels carry the biases and blind spots of the process that made them. SAM is a powerful general segmenter, not an omniscient one.

SAM brought the foundation-model paradigm — pre-train broadly, prompt for anything, generalize zero-shot — to a core computer-vision task, and proved it transfers beyond language. "Point at it and get a mask" is now a default primitive that countless tools and pipelines build on, from creative apps to scientific imaging.

Its most-copied idea may be the data engine: using an in-progress model to bootstrap the very dataset that trains it, in a compounding loop, to reach a scale humans could never label by hand. That model-in-the-loop annotation strategy is now a standard tool for building foundation models in data-scarce domains — arguably a bigger contribution than SAM's architecture itself.

The central limitation is named clearly: SAM has no semantics. It will cut out a thing without being able to say what the thing is, because no label was ever part of its training signal. It also misses fine structure — thin, wiry or highly detailed boundaries — can produce implausible mask edges at small scale, and the image encoder is far from real-time, so the interactivity is only for prompts after the first encode.

The dataset is the contribution with the longest reach and also the one that invites the most scrutiny: masks generated largely by a model are a training signal with the model's own biases folded into it, and the automatic stage's quality is asserted by sampled human review rather than exhaustive verification. That is a reasonable way to build a billion of anything, and it is a different epistemic object from a hand-annotated benchmark.

How it held up: promptable segmentation became a standard component rather than a research direction — the usual pattern is to pair SAM with something that supplies the missing semantics, either an open-vocabulary detector or a text-image model, so one system names the object and this one cuts it out. The data engine is the part that generalised furthest, and it is the same model-in-the-loop annotation loop that the rest of this site's data infrastructure pages describe. The missing-semantics gap points directly at the multimodal ladder, where the training signal is language from the start.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

From the network that started the deep-learning era to one you can segment anything with — and the shift from labels to self-supervision in between.

1. 1AlexNet2012
2. 2Vision Transformer2021
3. 3SimCLR2020
4. 4Masked Autoencoders2022
5. 5DINOv22023
6. 6Segment Anything2023

All 110 papers, in 19 ladders →