Masked autoencoders,
explained.

SimCLR's signal comes from comparing images, which means it needs a batch full of negatives and an augmentation pipeline tuned hard enough to close every shortcut. MAE's signal comes from a single image compared with itself: hide most of the patches and reconstruct the missing pixels. No pairs, no negatives, no batch-size requirement, and the augmentation reduces to random cropping.

That is BERT's objective moved to vision, and the interesting part is the number that had to change. BERT masks around 15% of tokens; MAE masks 75% of patches. The reason is redundancy: a missing word is often unrecoverable without understanding the sentence, whereas a missing image patch can usually be interpolated from its neighbours. At a low mask ratio the task is solvable by smoothing and teaches nothing about content. Three quarters removed makes interpolation useless.

Stated in units the ladder can compare: the high mask ratio is also what makes it cheap, through an asymmetry that is the paper's real engineering contribution. The encoder runs on the visible patches only — the masked ones are not fed to it at all, so the expensive network processes roughly a quarter of the sequence — and a deliberately small decoder reassembles the full set and predicts pixels for the masked positions only. Where SimCLR bought representation quality with batch size, MAE buys it with a large multiple in pre-training throughput at the same quality.

Self-supervised learning — training on raw data with no labels — transformed language through masked prediction: hide some words, predict them, and the model learns deep structure for free. The obvious dream was to do the same for vision: mask parts of an image, reconstruct them, get a powerful encoder without labels. But early attempts underwhelmed. Something about images made naive masking a weak teacher.

MAE diagnosed the mismatch and fixed it with three coupled choices: mask a very high fraction of the image, use an asymmetric encoder-decoder where the heavy encoder sees only visible patches, and reconstruct raw pixels. Together they turned masked image modeling from a curiosity into a state-of-the-art pre-training method — the image analogue of BERT it had been chasing.

Here is the key insight. Language is dense with meaning — every word matters, so BERT masks only ~15% and the task is already hard. Images are the opposite: hugely redundant. A missing patch of sky or grass is trivially guessed from its neighbors. Mask just a little and the model "solves" reconstruction by interpolating nearby pixels, learning nothing about objects, scenes, or structure.

MAE's answer is to mask 75% of the patches — enough that local interpolation no longer works. To fill in three-quarters of a scene from the remaining quarter, the model has to grasp what is actually in the image and how it fits together. The high mask ratio isn't a tuning detail; it is the mechanism that makes the task demand understanding rather than pixel-copying. And it doubles as an efficiency gift, because the encoder then has far fewer patches to process.

The architecture is deliberately lopsided. The encoder — a full Vision Transformer, the part you keep — runs only on the visible patches. The masked patches aren't fed in at all; they are simply dropped. A small, lightweight decoder then takes the encoded visible patches plus placeholder "mask tokens" for the missing ones and reconstructs the image. After training, the decoder is thrown away.

Because transformer self-attention scales with the square of the token count, encoding only the visible fraction is a big saving:

At a 75% mask (r = 0.75) the encoder sees a quarter of the patches, so its attention costs (1/4)² = 1/16 of processing them all — for 196 patches, 49 visible instead of 196. That is what lets MAE pre-train large encoders economically: the expensive part runs on a sliver of the input, and the cheap decoder handles the rest. That 1/16 is the attention-matrix saving alone; end to end the paper measures a smaller 2.8 to 4.1 times speedup in wall-clock training. The runnable version below computes the visible count and this attention speedup for any mask ratio.

The target is refreshingly plain: the decoder predicts the raw pixel values of each masked patch, and the loss is mean-squared error — but computed only on the masked patches. The visible patches, which the encoder already saw, are not scored; grading the model on copying what it was given would teach nothing.

Reconstructing pixels rather than higher-level features kept the objective simple and, somewhat surprisingly, worked very well. The reconstructions themselves can look blurry — the model isn't a great artist — but that was never the point. The blur is fine; the features the encoder learned on the way are what you take with you.

MAE pre-trains a plain Vision Transformer with no labels, no pairs and no negatives. The ablations use ViT-L/16 on ImageNet-1K images at 224×224 resolution, so each image is 196 patches.

1. Patchify and embed. The image is cut into non-overlapping 16×16 patches. Each is linearly projected to a token and given a fixed sine-cosine position embedding. A dummy token is appended to serve as a class token later; the method works about as well with average pooling instead.
2. Mask by shuffling. The token list is randomly shuffled and the last 75 percent is dropped. That is uniform sampling without replacement, so 49 tokens survive and 147 are removed. No sparse operations are needed, only a shuffle.
3. Encode only what is visible. The ViT encoder runs on the 49 visible tokens and nothing else. Mask tokens are never fed to it, so the expensive network does about a quarter of the per-token work and the attention matrix shrinks sixteenfold.
4. Rebuild the full sequence. A linear layer maps encoder outputs to the decoder's width. One shared, learned mask vector is appended for each removed patch, the list is unshuffled back into image order, and position embeddings are added to all 196 tokens. Without them the mask tokens would not know where they sit.
5. Decode. A small Transformer, by default 8 blocks of width 512 and under a tenth of the encoder's compute per token, processes all 196 tokens. A final linear layer outputs 16·16·3 = 768 pixel values per patch.
6. Score the masked patches only. The loss is mean squared error between predicted and true pixels over the 147 masked patches. In the best variant each target patch is first normalized by its own mean and standard deviation, which adds local contrast.
7. Train cheaply. Augmentation is just a random resized crop and flip. Color jitter hurt, and even no augmentation worked decently, because a fresh random mask each time already varies the task. Ablations run 800 epochs; the headline models run 1,600, and linear-probe accuracy had not saturated by then.
8. Drop the decoder. For recognition, the encoder processes all 196 patches of an uncorrupted image, either fine-tuned end to end or frozen under a linear classifier.

Why skipping the mask token matters

The tempting design feeds mask tokens through the encoder as BERT does. The paper tested it. Compute rose by 3.3 times, fine-tuning accuracy fell slightly and linear-probe accuracy fell by 14 points. The explanation is a gap between pre-training, where most of the encoder's input is mask tokens, and use, where none is. Leaving them out keeps the encoder's inputs real in both phases.

What a high mask ratio does

Fine-tuning accuracy was flat across masking ratios from 40 to 80 percent, but linear-probe accuracy climbed toward 75 percent. That is why 75 percent is the default, and well above the 15 percent BERT uses for words.

• V, M — the visible and masked patch index sets; N = 196 patches and ratio r = 0.75, so |V| = 49 and |M| = 147.
• W, pi — the shared patch projection and the fixed sine-cosine position embedding of patch i.
• P, m — the linear map to the decoder's width and the one shared learned mask vector.
• μi, σi — the mean and standard deviation of the pixels inside patch i; the target is each patch normalized by its own statistics.

Only masked patches enter the sum. At r = 0.75 the encoder's attention scores number 49² = 2,401 instead of 196² = 38,416, sixteen times fewer, while per-token layers shrink fourfold. At r = 0.25 the encoder would see 147 tokens, so scores would number 21,609, nine times as many as at 75 percent.

MAE is pre-trained on ImageNet-1K images without labels, then evaluated by fine-tuning the whole encoder or by training a linear classifier on frozen features. Fine-tuning top-1 accuracy, with the headline models pre-trained for 1,600 epochs:

The 87.8 used only ImageNet-1K data. Transfer carried over: on COCO detection, ViT-L reached 53.3 box AP against 49.3 for supervised pre-training, and on ADE20K segmentation 53.6 mIoU against 49.9.

Ablations on ViT-L, 800 epochs

Speed: with 8-block decoders, ViT-L trained 2.8 times faster and ViT-H 3.5 times faster than with mask tokens in the encoder. The honest caveat is linear probing: 73.5 is well behind fine-tuning, though tuning just the last block lifted ViT-L from 73.5 to 81.0.

The paper is straightforward about the trade that matters when choosing between rungs 3, 4 and 5: MAE's features are excellent when you fine-tune the whole encoder and comparatively weak under a linear probe, where contrastive methods do better. Both halves are real. Reconstruction teaches the model a great deal about image structure, but it does not organise the representation so that categories fall on either side of a hyperplane, because nothing in the objective ever asks it to compare two images.

It is also honest that pixel reconstruction is an odd target: a pixel-level loss spends capacity on texture and high-frequency detail that carries no semantic weight, and the paper discusses normalising the target to take some of that sting out rather than claiming pixels are the right thing to predict. Later work replaced the target with features from another network, which concedes the point.

How it held up: masked image modelling became a standard pre-training family, and the asymmetric encoder is the part that is copied without argument because it is close to free. The fine-tune-versus-probe split persisted as a real choice rather than being resolved — it is the reason rung 5 exists as a separate rung rather than a refinement, and the reason a practitioner picking a frozen backbone today usually picks rung 5's and one fine-tuning a detector often starts from this one.

MAE closed the gap between vision and language pre-training with a method that was simple, scalable, and effective — and it reframed a stalled idea by finding the one parameter that mattered. Masked image modeling had been tried; what MAE contributed was the realization that redundancy was the enemy, that a high mask ratio was the cure, and that the asymmetry it enabled made the whole thing efficient. Masked pre-training is now a standard tool across vision, and later multimodal and video models build directly on the recipe.

The takeaway generalizes past images: a self-supervised task is only as good as it is hard. If the model can cheat — copy a neighbor, interpolate, exploit redundancy — it learns the cheat, not the structure. Make the task genuinely require understanding, and rich representations follow. MAE's 75% is that principle turned into a number.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

From the network that started the deep-learning era to one you can segment anything with — and the shift from labels to self-supervision in between.

1. 1AlexNet2012
2. 2Vision Transformer2021
3. 3SimCLR2020
4. 4Masked Autoencoders2022
5. 5DINOv22023
6. 6Segment Anything2023

All 110 papers, in 19 ladders →