Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path →
Segment Anything,
explained.
Language got its foundation models — pre-train once, prompt for anything. Segmentation, the task of cutting out exactly which pixels belong to an object, was still stuck training a fresh model per dataset, per category. Segment Anything asked: what if you could just point at anything in any image and get a clean mask back, no training required? To pull it off, Meta didn't only build a model — it built a machine to build the dataset.
Contents 19 sections
The delta Segmentation, the old way The promptable segmentation task Three parts, split for speed Handling ambiguity gracefully The data engine: the real hero The limits Why it still matters How it works The math The evidence Segment Anything trade-offs: paying for the encoder once Common mistakes when building on Segment Anything Where Segment Anything shows up in production Since then Solve it yourself Questions Check yourself Where this sitsWhat changes against every rung below: a representation becomes an interface
Rungs 1 to 5 produce features, and something else has to be trained on top to do a task. SAM's change is to make the task itself promptable and general: given a point, a box or a rough mask, return the mask of the object that was indicated, zero-shot, for objects and domains never seen in training. The model is not a segmenter for a fixed label set; it is a function from a pointing gesture to a cut-out.
Two design choices follow from wanting that to be usable. The first is a split by cost: a heavy transformer image encoder runs once per image, and a very light prompt encoder and mask decoder run per click, so after the one-off encode each subsequent prompt is answered fast enough to feel interactive. The second is explicit handling of ambiguity — a click on a person's shirt might mean the shirt, the person, or the group — so the model emits several masks with confidence scores rather than averaging incompatible interpretations into one bad mask.
The units that actually explain the result are not architectural. SAM is trained on roughly a billion masks over about 11 million images, and that dataset did not exist; it was built by a data engine that ran in stages, with annotators correcting the model, then the model proposing and annotators only handling what it missed, then the model annotating automatically at scale. Each stage's model was trained on the previous stage's output. The model and the dataset bootstrapped each other, and that loop is the transferable idea on this rung.
Segmentation, the old way
Segmentation is pixel-level understanding: not just "there's a dog," but exactly which pixels are the dog, traced to its outline. It powers photo editing, medical imaging, robotics, self-driving — anywhere you need to isolate an object precisely rather than just draw a loose box.
The old workflow was narrow and expensive. You picked a fixed set of categories, collected a dataset with every object masked by hand (painstaking, pixel-perfect labeling), and trained a specialist model. It worked only for those categories, on that kind of image. A new domain meant a new dataset and a new model. Segmentation had never had its "foundation model" moment — a single pre-trained model you could point at anything.
The promptable segmentation task
SAM's reframing borrows from language models: make segmentation promptable. Instead of "segment these 80 fixed classes," the task becomes "given a prompt pointing at something, return its mask." The prompt is deliberately flexible — a single click on the object, a bounding box around it, a rough mask, even (in principle) text.
This one shift generalizes segmentation the way prompting generalized language. The model no longer needs a predefined vocabulary of objects; it responds to whatever you indicate. Train it to be good at "return the mask for this prompt" across enough examples, and it can segment novel objects it never saw a category label for — the definition of a foundation model for this task.
Three parts, split for speed
SAM is engineered around one practical goal: respond to prompts interactively, in real time. It splits into three pieces so the expensive work happens only once per image.
ViT · heavy · runs once
click / box · light
fast · masks out
The image encoder — a heavy Vision Transformer — processes the whole image into an embedding a single time. The lightweight prompt encoder turns a click or box into a small vector. The fast mask decoder then fuses the image embedding with the prompt to produce a mask in milliseconds. So after a one-time encode, you can click around an image and get instant masks — the responsiveness that made SAM feel magical.
Handling ambiguity gracefully
A single click is genuinely ambiguous. Click on someone's shirt — do you mean the shirt, the person wearing it, or the whole group they're standing in? There's no single correct answer, and a model that always guesses one will often guess wrong.
SAM's fix is to embrace it: for an ambiguous prompt, it outputs several valid masks at once — the shirt, the person, the group — each with a confidence score. The user (or a downstream system) picks the intended one. This multi-mask output turns ambiguity from a failure mode into a feature, and it's a big part of why SAM feels so usable in practice.
Ambiguity resolution by "propose a few good options, ranked" mirrors how humans clarify. It's a small design choice with outsized impact on how natural the model is to use interactively.
Mask quality is measured by IoU — the overlap between prediction and ground truth over their union — and that's what makes the multi-mask trick pay off:
A model forced to one answer is wrong whenever you meant a part; proposing shirt/person/group and ranking them means the right mask is always on offer. The runnable version below computes the IoU table and the pick.
The data engine: the real hero
A foundation model needs a foundation-scale dataset, and no billion-mask segmentation dataset existed — masks are far more expensive to label than image tags. So Meta built a data engine: a bootstrapping loop where model and data improve each other.
It ran in three gears. Assisted: annotators labeled masks with an early SAM helping, much faster than from scratch. Semi-automatic: SAM proposed masks for confident objects and humans filled the gaps. Fully automatic: SAM, now strong, blanketed each image with a grid of point prompts and generated masks on its own. Each round of new labels retrained a better SAM, which labeled more, faster. The output was SA-1B — over 1 billion masks across 11 million images, the largest segmentation dataset ever, most of it machine-generated but human-validated.
| Data engine stage | Who does the work |
|---|---|
| Assisted | humans, SAM helps |
| Semi-automatic | SAM proposes, humans fill gaps |
| Fully automatic | SAM labels the whole image |
The limits
SAM cuts out masks but doesn't name them — it tells you where an object is, not what it is. Semantic labels need a separate model. Its heavy ViT image encoder is compute-hungry, making the original SAM impractical on-device or for real-time video (later work like MobileSAM and SAM 2 tackled speed and video). And on very fine or thin structures, its masks can miss detail.
Text prompting, teased in the paper, was more proof-of-concept than robust feature. And like any model, SA-1B's automatically-generated labels carry the biases and blind spots of the process that made them. SAM is a powerful general segmenter, not an omniscient one.
Why it still matters
SAM brought the foundation-model paradigm — pre-train broadly, prompt for anything, generalize zero-shot — to a core computer-vision task, and proved it transfers beyond language. "Point at it and get a mask" is now a default primitive that countless tools and pipelines build on, from creative apps to scientific imaging.
Its most-copied idea may be the data engine: using an in-progress model to bootstrap the very dataset that trains it, in a compounding loop, to reach a scale humans could never label by hand. That model-in-the-loop annotation strategy is now a standard tool for building foundation models in data-scarce domains — arguably a bigger contribution than SAM's architecture itself.
SAM's image encoder is a Vision Transformer; its foundation-model spirit echoes CLIP. Together they show how the pre-train-and-prompt paradigm reshaped vision.
How SAM encodes, decodes and is trained, step by step
SAM splits the work by cost. One heavy network summarizes the image once; a very light network turns each click, box or rough mask into masks against that summary. The sequence below follows one image and one prompt.
- Encode the image once. The image is rescaled and padded to 1024×1024 and passed through a ViT-H/16 initialized from MAE pre-training. It uses 14×14 windowed attention with four evenly spaced global-attention blocks to keep high resolution affordable. The output is a 64×64 embedding with 256 channels, reused for every prompt.
- Encode the prompt. A point becomes a positional encoding of its location plus one of two learned vectors saying foreground or background. A box becomes two such corner embeddings. Text goes through CLIP's text encoder, and a rough mask prompt passes through convolutions and is added element by element to the image embedding.
- Add output tokens. The decoder starts with the prompt embeddings plus learned output tokens: one for each of three masks and one for the predicted IoU, a score for how well each mask matches the true object.
- Run two-way decoder blocks. Each block does self-attention among the tokens, cross-attention from tokens to the image embedding, an MLP on the tokens, and cross-attention from the image embedding back to the tokens. Information flows in both directions, so the image features are also updated by the prompt. Positional encodings are re-added at every attention layer. There are two such blocks.
- Produce the masks. The image embedding is upsampled with transposed convolutions. An MLP turns each mask token into a small classifier, which is applied at every location of the upsampled embedding to give a foreground probability. A separate head reads the IoU token and predicts one confidence score per mask.
- Train against the best guess. For a single ambiguous prompt, only the lowest-loss of the three predicted masks receives a gradient, so different tokens can specialize in whole object, part and subpart. A fourth mask is used when several prompts are given, which makes the ambiguity much smaller; it is the only one returned in that case.
- Simulate clicking. Training starts from a random foreground point or a noisy box. The model predicts, and the next point is drawn from the error region between prediction and truth, as foreground where mask was missed and background where it overreached. The previous mask logits are fed back as a prompt. Over 11 rounds that is one initial prompt, eight corrective points and two rounds with no new input so the model learns to refine its own mask.
- Run it for real. SAM trained for 68 hours on 256 A100 GPUs. Given a stored image embedding, the prompt encoder and decoder answer in about 50 milliseconds, even in a browser on a CPU.
Why the split works
The decoder uses under 1 percent of the image encoder's compute, which is why training could afford eleven prompt rounds per mask and why a person can click repeatedly without re-encoding. The cost of the encoder is paid once per image and amortized over every prompt.
For whole-image masks, the model is prompted with a regular grid of points, each returning up to three masks. Masks are kept if their predicted IoU is high and if they are stable, meaning thresholding the probability map slightly above and below one half gives a similar mask; duplicates are removed by non-maximal suppression.
SAM's loss: focal, dice and a predicted IoU
mk— the k-th predicted mask for one prompt;gis the ground-truth mask;k*is the prediction with the lowest mask loss, the only one that receives gradient.Lfocal— a focal loss that down-weights easy pixels, weighted 20 times the dice loss.ŝk— the IoU head's estimate for mask k, trained with squared error against the mask's true IoU.
A worked case: a predicted mask of 60 pixels, a truth of 80 pixels and an overlap of 48 give a union of 92, so IoU = 48 / 92 ≈ 0.52, and dice = 96 / 140 ≈ 0.69, a dice loss of about 0.31. With three predictions whose losses are 0.9, 0.3 and 0.6, only the 0.3 mask is trained on this step, which lets the other two tokens learn to cover other readings of the click.
What SAM measured: the data engine, mask quality and zero-shot transfer
The result is as much a dataset as a model. The data engine ran in three stages, each retraining the model on what the last one collected.
| Stage | Images | Masks | What changed |
|---|---|---|---|
| Assisted-manual | 120k | 4.3M | Time per mask fell from 34 to 14 seconds; 20 to 44 masks per image |
| Semi-automatic | 180k | 5.9M more | Detector pre-fills confident masks; 44 to 72 per image |
| Fully automatic | 11M | 1.1B | Grid of points, no annotators; 99.1% of final masks |
At 14 seconds per mask, assisted labeling was 6.5 times faster than COCO annotation. To check the automatic masks, 500 images (about 50,000 masks) were re-labeled by professionals; 94 percent of pairs overlapped by more than 90 IoU, 97 percent by more than 75, against 85 to 91 for agreement between human annotators in earlier work. Training on automatic masks alone was nearly as good as training on all of them.
Zero-shot transfer and ablations
- 23 datasets. With one center point as the prompt, SAM beat the interactive baseline RITM on 16 of 23 datasets, by as much as 47 IoU. With an oracle picking the best of its three masks it won on all 23, which shows how much of the gap on the others is ambiguity.
- Human ratings. Annotators rated SAM's masks consistently above RITM's. A single-output SAM was rated lower.
- Proposals. Producing multiple masks per point helped recall when generating object proposals.
- Other tasks. Edge detection, proposals, instance segmentation and text-to-mask all worked without training on them, with text prompts preliminary.
- Cost. The released model took 68 hours on 256 A100 GPUs.
Segment Anything trade-offs: paying for the encoder once
SAM's economics rest on amortization. The ViT-H encoder on a 1024×1024 input is the whole bill; each prompt after that costs under 1% of it, about 50 milliseconds on a CPU. That is a bargain when a person clicks many times on one image and a poor one when every image gets one prompt, since then you pay the full encoder per mask. Automatic whole-image mode is costlier still: a 32×32 grid with three masks per point is 3,072 candidates to score, filter and de-duplicate. Lighter released encoders, ViT-B and ViT-L, trade mask quality for speed.
| Option | Names the object? | New categories | Cost profile |
|---|---|---|---|
| SAM, interactive | No | Any, by pointing | One heavy encode, then cheap prompts |
| SAM, automatic grid | No | Any | One encode plus thousands of decodes and filtering |
| Closed-set instance segmenter | Yes, from a fixed list | Needs new labels and retraining | One pass per image |
| Open-vocabulary detector plus SAM | Yes, from text | Any that the detector can name | Two models; the detector's boxes become SAM's prompts |
Common mistakes when building on Segment Anything
- Re-encoding the image for every prompt
In the official package,
SamPredictor.set_imagecomputes the embedding andpredictanswers prompts against it. Calling both per click pays the heavy encoder every time and throws away the design's whole point. - Taking the first mask
For a single ambiguous click, ask for all three masks and rank them by the predicted IoU score; mask order is not a ranking. With several points or a box, ask for one mask, since the extra prompts remove the ambiguity.
- Passing prompts in the wrong coordinate frame
The image is resized so its long side is 1024 and then padded. The high-level predictor transforms point and box coordinates to match; the raw model expects them already transformed, and untransformed points land on the wrong pixels without any error.
- Reading the predicted IoU as measured accuracy
The score is the model's own estimate, learned on SA-1B-like photos. On a domain far from that, such as microscopy or medical scans, it can be confidently high for a wrong mask, so check against labelled samples.
- Expecting fine boundaries from the decoder
Masks are predicted at 256×256, four times the 64×64 embedding, and upsampled to the image. Thin structures and hair-fine edges are lost before upsampling, so add refinement if edges matter.
Where Segment Anything shows up in production
- Click-to-segment in the browser
The official repository exports the prompt encoder and mask decoder to ONNX, so they can run client-side while a server computes the image embedding once. That split is how interactive labelling and editing tools get instant masks.
- Paired with something that names things
Grounded-SAM feeds boxes from the Grounding DINO open-vocabulary detector into SAM as prompts, so text picks the object and SAM cuts it out.
- Video
SAM 2 extends promptable segmentation to video, keeping a memory of earlier frames so a click on one frame tracks the object through the clip.
- In interviews
An interactive segmentation service is a good design prompt: cache the embedding per image, run the light decoder near the user, and pair it with a detector or classifier for labels.
What the authors admit, and what happened next
The central limitation is named clearly: SAM has no semantics. It will cut out a thing without being able to say what the thing is, because no label was ever part of its training signal. It also misses fine structure — thin, wiry or highly detailed boundaries — can produce implausible mask edges at small scale, and the image encoder is far from real-time, so the interactivity is only for prompts after the first encode.
The dataset is the contribution with the longest reach and also the one that invites the most scrutiny: masks generated largely by a model are a training signal with the model's own biases folded into it, and the automatic stage's quality is asserted by sampled human review rather than exhaustive verification. That is a reasonable way to build a billion of anything, and it is a different epistemic object from a hand-annotated benchmark.
How it held up: promptable segmentation became a standard component rather than a research direction — the usual pattern is to pair SAM with something that supplies the missing semantics, either an open-vocabulary detector or a text-image model, so one system names the object and this one cuts it out. The data engine is the part that generalised furthest, and it is the same model-in-the-loop annotation loop that the rest of this site's data infrastructure pages describe. The missing-semantics gap points directly at the multimodal ladder, where the training signal is language from the start.
Score masks and resolve an ambiguous click
Segmentation quality is one formula — IoU, the overlap between a predicted mask and the truth over their union — and SAM's ambiguity trick is what that formula rewards. This scores three nested masks (shirt ⊂ person ⊂ group) against every possible intent, then shows the payoff: click the shirt and a single "whole object" model scores a weak 0.33, while SAM's multi-mask output offers the shirt at a perfect 1.00. Whichever part you meant, the right mask is on the menu. Change the masks or the intent.
Write the body of best_iou. Everything else in the file is
already written; this is the function the paper is about. Press ▶ Run to check it against
the 5 assertions below — they start red.
- IoU of a mask with itself is 1
- IoU of disjoint masks is 0
- IoU is symmetric
- A nested mask overlaps partially (0 < IoU < 1)
- Multi-mask always offers the intended mask where single-mask misses
Questions people ask about Segment Anything
How does SAM's mask decoder work?
The decoder is a small Transformer with two blocks. Its inputs are the prompt embeddings, four learned output tokens (three for masks, one for IoU) and the image embedding. Each block applies self-attention among tokens, cross-attention from tokens to the image, an MLP on the tokens, and cross-attention from the image back to the tokens, so information moves both ways. Afterward the image embedding is upsampled, and an MLP turns each mask token into a per-location classifier that outputs foreground probabilities. The whole decoder uses under 1 percent of the image encoder's compute and runs in about 50 milliseconds on a CPU.
Why does SAM output three masks, and how is that trained?
A single click can mean a part, an object or a group, and a model forced to give one answer tends to average incompatible readings into a blurry mask. SAM outputs three masks, enough to cover whole, part and subpart in most nested cases. During training only the mask with the lowest loss receives gradient, so the three tokens can specialize. A separate head predicts an IoU score for each mask so they can be ranked. When several prompts are given, ambiguity shrinks, so a fourth mask token is used and only that mask is returned.
What loss function does SAM use to train mask prediction?
Mask prediction uses a weighted sum of focal loss and dice loss in a 20 to 1 ratio. Focal loss emphasizes hard pixels, which matters because most pixels are easy background, while dice loss measures overlap directly. A separate IoU head is trained with squared error between its predicted score and the mask's actual IoU with the ground truth, added with weight 1.0. Only the best of the three candidate masks contributes to the mask loss. The authors found that extra supervision after every decoder layer was unhelpful, so loss is applied at the end only.
How did SAM's data engine produce the billion masks in SA-1B?
In three stages. Professional annotators first labeled masks with browser-based SAM assistance, collecting 4.3 million masks on 120,000 images while time per mask fell from 34 to 14 seconds. In the semi-automatic stage a detector trained on those masks pre-filled confident objects and annotators added the rest, giving 5.9 million more. In the final stage the ambiguity-aware model was prompted with a 32 by 32 grid of points on all 11 million images, producing 1.1 billion masks, 99.1 percent of the total. Professional re-labeling of a sample put 94 percent of pairs above 90 IoU.
How does SAM generate masks for a whole image without any prompt?
It prompts itself with a regular grid of points, 32 by 32 in the data engine. Each point returns up to three masks. A mask is kept only if its predicted IoU is confident and if it is stable, meaning that thresholding the probability map at one half plus or minus a small value gives a similar result. Non-maximal suppression then removes duplicates, and overlapping zoomed-in crops improve small masks. Because a point on a part returns the part, the object and sometimes the whole, the grid yields masks at several scales, which is what makes the approach work for generating proposals.
4 questions to test what you just read
Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.
A predicted mask has 50 pixels, the truth has 100, and 40 overlap. What are the IoU and the dice loss?
Show answer
Union = 50 + 100 − 40 = 110, so IoU = 40 / 110 ≈ 0.36. Dice = 2 × 40 / (50 + 100) ≈ 0.53, so the dice loss is about 0.47. Dice is more forgiving than IoU here, which is one reason the two are used together.
The three candidate masks for a click have losses 0.8, 0.2 and 0.5. Which receives gradient, and why is that useful?
Show answer
Only the 0.2 mask. The others are left free, so different tokens can learn other readings of an ambiguous click, such as part versus whole, instead of all being pulled toward one averaged mask.
A 32×32 grid of points with three masks per point yields how many candidates per image before filtering, and which filters thin them out?
Show answer
32 × 32 × 3 = 3,072 candidate masks. Predicted IoU confidence and the stability check discard unreliable ones, and non-maximal suppression removes duplicates, typically leaving far fewer masks than the raw 3,072.
Why can training afford 11 prompt rounds per mask while the image is encoded only once?
Show answer
The prompt encoder and decoder together need under 1 percent of the encoder's compute. Each extra round adds little cost, and the image embedding is computed once and reused across all 11.
Learning to see
From the network that started the deep-learning era to one you can segment anything with — and the shift from labels to self-supervision in between.
- 1AlexNet2012
- 2Vision Transformer2021
- 3SimCLR2020
- 4Masked Autoencoders2022
- 5DINOv22023
- 6Segment Anything2023
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Paper Breakdowns
- Attention Is All You NeedThe paper that introduced the Transformer — the architecture underneath GPT, BERT, Claude, and Gemini. Self-attention, Query/Key/Value, multi-head attention, positional encoding, and the real BLEU results, explained in depth with analogies and animated visuals, not just equations.Read →
- Word2VecHow words became math. Dense word vectors learned from the company a word keeps, so king − man + woman lands near queen. Skip-gram, CBOW, the distributional hypothesis, and the origin of every embedding model powering search and RAG.Read →
- ResNetThe skip connection that made deep learning deep. Why stacking layers made networks worse, and how residual learning let them reach 150+ layers and win ImageNet — the identity shortcut every Transformer still uses inside every block.Read →
- Vision Transformer (ViT)When an image became 16×16 words. Cut a picture into patches, treat each patch as a token, and feed them to a plain Transformer — no convolutions. Patch embeddings, the class token, and why data scale lets attention beat the CNNs that ruled computer vision for a decade.Read →
- Latent DiffusionThe paper behind Stable Diffusion. Compress an image into a small latent space with a VAE, run the diffusion denoising there instead of on pixels for a tens-of-times efficiency win, and steer it with text via cross-attention. The architecture that put text-to-image generation on consumer GPUs.Read →
- DDPMThe original denoising diffusion math — the paper that made diffusion work. A fixed forward process adds Gaussian noise over many steps; a U-Net learns the reverse. The masterstroke: train the network to simply predict the added noise with a plain MSE loss. The foundation under every modern image generator.Read →