Flamingo,
explained.

By 2022 the field had superb pretrained pieces: vision encoders like CLIP that turn images into rich features, and large language models fluent in text. The obvious goal was a model that could do both — read an image and reason about it in words. The expensive way is to train one giant multimodal model from scratch. Flamingo took the cheap, elegant route: keep both pretrained models frozen and train only a small connector between them.

Freezing is the whole strategy. It preserves the language model's hard-won abilities and the vision encoder's perception, and it makes training far cheaper — only the bridge learns. But it raises two engineering questions. Vision encoders emit a variable, often large number of features per image (and far more per video); how do you feed that into a language model expecting a tidy token stream? And how do you inject visual information into a frozen model without disrupting the delicate weights that make it work? Flamingo answers both.

The first component solves the variable-size problem. A vision encoder might produce hundreds of feature vectors for one image, thousands for a video — and that count changes with resolution and length. The Perceiver Resampler maps any such set to a fixed, small number of latent tokens. It does this with cross-attention: a fixed set of learned query vectors (say 64 of them) attend to the visual features and absorb them, producing exactly 64 output tokens no matter how many features came in.

Because M is fixed, the cost of everything downstream — feeding vision into the language model — is constant, independent of image resolution or video length. A 576-feature image and a 2,304-feature one both become 64 tokens; the language model sees the same tidy, bounded input either way. The runnable version below confirms the fixed output and the constant downstream cost.

Now the delicate part: inject those visual tokens into a frozen language model without wrecking it. Flamingo interleaves new gated cross-attention layers between the language model's existing (frozen) layers. Each new layer lets the text tokens attend to the visual tokens — but its output is multiplied by a learnable tanh(α) gate, with α initialized to zero:

At initialization tanh(0) = 0, so the cross-attention contributes exactly nothing and the model is bit-for-bit the original language model — untouched, unbroken. As training proceeds, the gates α gradually open, smoothly letting visual information flow in. This zero-init trick is what makes adding a modality to a frozen model safe: you begin from a known-good model and only ever add, never disrupt. The runnable version below shows the gate starting at 0 and opening.

To get few-shot multimodal ability, Flamingo trains on interleaved image-and-text sequences — web pages and documents where images and words are naturally mixed, images appearing inline with the text about them. The model learns to predict the text given the interspersed images, with attention masking so each text span attends to the images that precede it.

Training on interleaved data is what teaches Flamingo to handle a prompt like "here's an image and its caption; here's a second; now caption this third image" — the multimodal analogue of in-context few-shot prompting. The same interleaving that appears in training appears at inference as the prompt format.

Flamingo's headline was few-shot multimodal learning — the ability GPT-3 brought to text, now for images:

The few-shot result mattered because it showed multimodal models could be prompted like language models, rather than requiring a fine-tune per task. That flexibility — describe the task with examples, get an answer — is exactly what made LLMs so useful, extended to vision.

Flamingo's architecture — freeze strong pretrained encoders, train a lightweight connector — became a dominant template for visual language models. The specific pieces recur throughout the field: resampler-style compression of visual tokens (and its cousins, like Q-Former in BLIP-2), and lightweight connectors that graft vision onto a frozen or lightly-tuned LLM (as in CLIP-plus-projector systems). The zero-initialized gate — adding a new capability by starting from a no-op — shows up wherever you want to extend a pretrained model safely.

Its deeper lesson is about composition. Rather than train one monolithic model to do everything, Flamingo showed you can take excellent specialist models and stitch them together with a small learned interface, inheriting both their strengths at a fraction of the cost. As pretrained components proliferate, that compositional approach — bridge, don't rebuild — has only grown more central to how multimodal systems are built.