Segment Anything,
explained.

Segmentation is pixel-level understanding: not just "there's a dog," but exactly which pixels are the dog, traced to its outline. It powers photo editing, medical imaging, robotics, self-driving — anywhere you need to isolate an object precisely rather than just draw a loose box.

The old workflow was narrow and expensive. You picked a fixed set of categories, collected a dataset with every object masked by hand (painstaking, pixel-perfect labeling), and trained a specialist model. It worked only for those categories, on that kind of image. A new domain meant a new dataset and a new model. Segmentation had never had its "foundation model" moment — a single pre-trained model you could point at anything.

SAM's reframing borrows from language models: make segmentation promptable. Instead of "segment these 80 fixed classes," the task becomes "given a prompt pointing at something, return its mask." The prompt is deliberately flexible — a single click on the object, a bounding box around it, a rough mask, even (in principle) text.

This one shift generalizes segmentation the way prompting generalized language. The model no longer needs a predefined vocabulary of objects; it responds to whatever you indicate. Train it to be good at "return the mask for this prompt" across enough examples, and it can segment novel objects it never saw a category label for — the definition of a foundation model for this task.

SAM is engineered around one practical goal: respond to prompts interactively, in real time. It splits into three pieces so the expensive work happens only once per image.

The image encoder — a heavy Vision Transformer — processes the whole image into an embedding a single time. The lightweight prompt encoder turns a click or box into a small vector. The fast mask decoder then fuses the image embedding with the prompt to produce a mask in milliseconds. So after a one-time encode, you can click around an image and get instant masks — the responsiveness that made SAM feel magical.

A single click is genuinely ambiguous. Click on someone's shirt — do you mean the shirt, the person wearing it, or the whole group they're standing in? There's no single correct answer, and a model that always guesses one will often guess wrong.

SAM's fix is to embrace it: for an ambiguous prompt, it outputs several valid masks at once — the shirt, the person, the group — each with a confidence score. The user (or a downstream system) picks the intended one. This multi-mask output turns ambiguity from a failure mode into a feature, and it's a big part of why SAM feels so usable in practice.

Mask quality is measured by IoU — the overlap between prediction and ground truth over their union — and that's what makes the multi-mask trick pay off:

A model forced to one answer is wrong whenever you meant a part; proposing shirt/person/group and ranking them means the right mask is always on offer. The runnable version below computes the IoU table and the pick.

A foundation model needs a foundation-scale dataset, and no billion-mask segmentation dataset existed — masks are far more expensive to label than image tags. So Meta built a data engine: a bootstrapping loop where model and data improve each other.

It ran in three gears. Assisted: annotators labeled masks with an early SAM helping, much faster than from scratch. Semi-automatic: SAM proposed masks for confident objects and humans filled the gaps. Fully automatic: SAM, now strong, blanketed each image with a grid of point prompts and generated masks on its own. Each round of new labels retrained a better SAM, which labeled more, faster. The output was SA-1B — over 1 billion masks across 11 million images, the largest segmentation dataset ever, most of it machine-generated but human-validated.

Trained on SA-1B, SAM segments objects zero-shot — cutting out things it was never explicitly trained to recognize, across wildly different image types, from a simple click. It matched or beat many specialist, fully-supervised segmentation models on datasets it had never seen, the hallmark of a true foundation model.

Meta released the model, the code, and the entire SA-1B dataset. Because SAM produces reusable masks from cheap prompts, it slotted in as a building block everywhere: image and video editing, medical and satellite imaging, robotics, and as the segmentation front-end for larger multimodal systems. It became infrastructure almost immediately.

SAM cuts out masks but doesn't name them — it tells you where an object is, not what it is. Semantic labels need a separate model. Its heavy ViT image encoder is compute-hungry, making the original SAM impractical on-device or for real-time video (later work like MobileSAM and SAM 2 tackled speed and video). And on very fine or thin structures, its masks can miss detail.

Text prompting, teased in the paper, was more proof-of-concept than robust feature. And like any model, SA-1B's automatically-generated labels carry the biases and blind spots of the process that made them. SAM is a powerful general segmenter, not an omniscient one.

SAM brought the foundation-model paradigm — pre-train broadly, prompt for anything, generalize zero-shot — to a core computer-vision task, and proved it transfers beyond language. "Point at it and get a mask" is now a default primitive that countless tools and pipelines build on, from creative apps to scientific imaging.

Its most-copied idea may be the data engine: using an in-progress model to bootstrap the very dataset that trains it, in a compounding loop, to reach a scale humans could never label by hand. That model-in-the-loop annotation strategy is now a standard tool for building foundation models in data-scarce domains — arguably a bigger contribution than SAM's architecture itself.