Masked autoencoders,
explained.

Self-supervised learning — training on raw data with no labels — transformed language through masked prediction: hide some words, predict them, and the model learns deep structure for free. The obvious dream was to do the same for vision: mask parts of an image, reconstruct them, get a powerful encoder without labels. But early attempts underwhelmed. Something about images made naive masking a weak teacher.

MAE diagnosed the mismatch and fixed it with three coupled choices: mask a very high fraction of the image, use an asymmetric encoder-decoder where the heavy encoder sees only visible patches, and reconstruct raw pixels. Together they turned masked image modeling from a curiosity into a state-of-the-art pre-training method — the image analogue of BERT it had been chasing.

Here is the key insight. Language is dense with meaning — every word matters, so BERT masks only ~15% and the task is already hard. Images are the opposite: hugely redundant. A missing patch of sky or grass is trivially guessed from its neighbors. Mask just a little and the model "solves" reconstruction by interpolating nearby pixels, learning nothing about objects, scenes, or structure.

MAE's answer is to mask 75% of the patches — enough that local interpolation no longer works. To fill in three-quarters of a scene from the remaining quarter, the model has to grasp what is actually in the image and how it fits together. The high mask ratio isn't a tuning detail; it is the mechanism that makes the task demand understanding rather than pixel-copying. And it doubles as an efficiency gift, because the encoder then has far fewer patches to process.

The architecture is deliberately lopsided. The encoder — a full Vision Transformer, the part you keep — runs only on the visible patches. The masked patches aren't fed in at all; they are simply dropped. A small, lightweight decoder then takes the encoded visible patches plus placeholder "mask tokens" for the missing ones and reconstructs the image. After training, the decoder is thrown away.

Because transformer self-attention scales with the square of the token count, encoding only the visible fraction is a big saving:

At a 75% mask (r = 0.75) the encoder sees a quarter of the patches, so its attention costs (1/4)² = 1/16 of processing them all — for 196 patches, 49 visible instead of 196. That is what lets MAE pre-train large encoders economically: the expensive part runs on a sliver of the input, and the cheap decoder handles the rest. The runnable version below computes the visible count and this speedup for any mask ratio.

The target is refreshingly plain: the decoder predicts the raw pixel values of each masked patch, and the loss is mean-squared error — but computed only on the masked patches. The visible patches, which the encoder already saw, are not scored; grading the model on copying what it was given would teach nothing.

Reconstructing pixels rather than higher-level features kept the objective simple and, somewhat surprisingly, worked very well. The reconstructions themselves can look blurry — the model isn't a great artist — but that was never the point. The blur is fine; the features the encoder learned on the way are what you take with you.

The point of a pre-training method is not the reconstructions — it is how well the encoder transfers. MAE delivered:

Notably, MAE needed none of the heavy data augmentation that contrastive self-supervised methods rely on. Masking is the augmentation — a different random 75% each time — which is part of why the recipe is so clean.

MAE closed the gap between vision and language pre-training with a method that was simple, scalable, and effective — and it reframed a stalled idea by finding the one parameter that mattered. Masked image modeling had been tried; what MAE contributed was the realization that redundancy was the enemy, that a high mask ratio was the cure, and that the asymmetry it enabled made the whole thing efficient. Masked pre-training is now a standard tool across vision, and later multimodal and video models build directly on the recipe.

The takeaway generalizes past images: a self-supervised task is only as good as it is hard. If the model can cheat — copy a neighbor, interpolate, exploit redundancy — it learns the cheat, not the structure. Make the task genuinely require understanding, and rich representations follow. MAE's 75% is that principle turned into a number.