Flamingo,
explained.

Rung 1 puts images and text in one space, which supports matching and retrieval and not generation — it can tell you which of several captions fits, and it cannot write one. What you want is a language model that can see, and the obvious route is to train one on images and text together from scratch, which is enormously expensive and risks a model worse at language than the language models already available.

The change is to keep the language model frozen and add sight around it, which preserves everything it knows at zero risk. Two mechanisms make that work. A resampler maps however many visual features an image or a run of video frames produces down to a fixed, small number of tokens, so visual input has a constant cost regardless of resolution or frame count. And cross-attention layers are inserted between the frozen model's existing layers so it can read those tokens.

The detail worth carrying up the ladder is how those inserted layers avoid damaging the model they are inserted into. Each is wrapped in a gate initialised at exactly zero, so at the start of training the inserted layer contributes nothing and the network computes bit-for-bit what the original language model computed. Vision is then faded in as the gate opens, rather than arriving as a perturbation that has to be recovered from. Training on interleaved image-and-text documents rather than isolated pairs is what then gives it few-shot behaviour: show it two labelled examples and a third image, and it continues the pattern.

By 2022 the field had superb pretrained pieces: vision encoders like CLIP that turn images into rich features, and large language models fluent in text. The obvious goal was a model that could do both — read an image and reason about it in words. The expensive way is to train one giant multimodal model from scratch. Flamingo took the cheap, elegant route: keep both pretrained models frozen and train only a small connector between them.

Freezing is the whole strategy. It preserves the language model's hard-won abilities and the vision encoder's perception, and it makes training far cheaper — only the bridge learns. But it raises two engineering questions. Vision encoders emit a variable, often large number of features per image (and far more per video); how do you feed that into a language model expecting a tidy token stream? And how do you inject visual information into a frozen model without disrupting the delicate weights that make it work? Flamingo answers both.

The first component solves the variable-size problem. A vision encoder might produce hundreds of feature vectors for one image, thousands for a video — and that count changes with resolution and length. The Perceiver Resampler maps any such set to a fixed, small number of latent tokens. It does this with cross-attention: a fixed set of learned query vectors (say 64 of them) attend to the visual features and absorb them, producing exactly 64 output tokens no matter how many features came in.

Because M is fixed, the cost of everything downstream — feeding vision into the language model — is constant, independent of image resolution or video length. A 576-feature image and a 2,304-feature one both become 64 tokens; the language model sees the same tidy, bounded input either way. The runnable version below confirms the fixed output and the constant downstream cost.

Now the delicate part: inject those visual tokens into a frozen language model without wrecking it. Flamingo interleaves new gated cross-attention layers between the language model's existing (frozen) layers. Each new layer lets the text tokens attend to the visual tokens — but its output is multiplied by a learnable tanh(α) gate, with α initialized to zero:

At initialization tanh(0) = 0, so the cross-attention contributes exactly nothing and the model is bit-for-bit the original language model — untouched, unbroken. As training proceeds, the gates α gradually open, smoothly letting visual information flow in. This zero-init trick is what makes adding a modality to a frozen model safe: you begin from a known-good model and only ever add, never disrupt. The runnable version below shows the gate starting at 0 and opening.

To get few-shot multimodal ability, Flamingo trains on interleaved image-and-text sequences — web pages and documents where images and words are naturally mixed, images appearing inline with the text about them. The model learns to predict the text given the interspersed images, with attention masking so each text span cross-attends only to the most recent image that precedes it.

Training on interleaved data is what teaches Flamingo to handle a prompt like "here's an image and its caption; here's a second; now caption this third image" — the multimodal analogue of in-context few-shot prompting. The same interleaving that appears in training appears at inference as the prompt format.

Flamingo maps any interleaved run of images, videos and text to text. Two frozen pretrained networks do most of the work, the trainable parts sit between them, and a mask decides which picture each word may look at, which is the detail most descriptions of Flamingo leave out.

1. Freeze a vision encoder. An NFNet-F6 image encoder is pretrained contrastively on image–text pairs, CLIP-style, then frozen. Its final feature grid is flattened into a set of vectors. For video, frames are sampled at one per second, encoded independently, tagged with learned temporal embeddings and flattened into one set.
2. Resample to 64 tokens. In the Perceiver Resampler, 64 learned latent queries cross-attend to the visual features through a few Transformer layers. Unusually, the keys and values are built from the visual features concatenated with the latents themselves, which the authors found worked slightly better. The output is 64 visual tokens per image or per video clip.
3. Mark the images in the text. An <image> tag replaces each image at its position in the text, and an end-of-chunk token is added before every image and at the end of the document.
4. Give each word one image. For each text token ℓ, φ(ℓ) is the index of the last image that appears before it, and the cross-attention mask lets token ℓ read only that image's 64 tokens; text before the first image reads none. Earlier images still reach later words, indirectly, through the language model's own self-attention over the text that followed them.
5. Insert gated blocks. New GATED XATTN-DENSE blocks sit between the frozen language-model layers: a cross-attention sublayer, with queries from the text and keys and values from the visual tokens, then a feed-forward sublayer, each added residually after multiplication by tanh of its own learned scalar, which starts at 0. Flamingo-3B adds a block before every layer, Flamingo-9B every fourth layer and Flamingo-80B every seventh.
6. Choose the language models. Frozen Chinchilla models of 1.4B, 7B and 70B parameters give Flamingo-3B, -9B and -80B. "Small connector" is relative: in Flamingo-80B the new gated layers come to about 10B trainable parameters, plus a resampler of about 194M.
7. Train on a weighted mixture. M3W, text and images extracted from about 43 million web pages, sampled as 256-token chunks with at most 5 images; ALIGN, 1.8 billion images with alt text; LTIP, 312 million images with longer descriptions; and VTP, 27 million short videos with descriptions, about 22 seconds long on average. The loss is a weighted sum of each dataset's next-token loss on the text, with gradients from all four accumulated in every update.
8. Prompt it at test time. A prompt is a few support examples, each an image followed by its text (a caption, or a question and its answer), then the query image. Open-ended answers come from beam search; multiple-choice tasks are scored by each option's log-likelihood. Support examples can be picked with RICES, which retrieves the pool examples whose images look most like the query.

• yℓ — the ℓ-th text token; only text is ever predicted, never images
• x≤ℓ — the images and videos that appear before token ℓ in the sequence
• φ(ℓ) — the index of the last image before ℓ, or 0 if there is none; Rφ(ℓ) is that image's 64 resampler tokens
• α, α′ — one learned scalar for each sublayer of each inserted block, both initialized at 0
• λm — the weight of dataset m among M3W, ALIGN, LTIP and VTP

At α = α′ = 0, tanh(0) = 0 and both new sublayers add nothing, so p(y | x) is exactly the frozen language model's probability for the text. Gradients still flow: the derivative of tanh at 0 is 1, so at the first step the gate scalars receive gradient while the cross-attention weights behind them receive none. Large |α| saturates tanh at ±1, so a gate can pass its sublayer's full output but never amplify it. And although each cross-attention reads only R_φ(ℓ), the likelihood still conditions on every earlier image, which reaches token ℓ through self-attention.

All three sizes were tested on 16 image and video benchmarks covering captioning, visual question answering, dialogue and classification, prompted with 0 to 32 examples and fine-tuned only where stated. Design choices were made on 5 development benchmarks (COCO, OK-VQA, VQAv2, MSVD-QA, VATEX); the other 11 were never used for design decisions, so they give unbiased few-shot estimates. The largest run used 1,536 TPU chips for 15 days.

The ablations train Flamingo-3B variants and report one overall score across the development benchmarks; the full model scores 70.7.

The interleaved web data matters most; without it the model loses the largest share of its score in the whole study. Without the zero-initialized gate, training became unstable. Unfreezing the language model hurt whether it was fine-tuned or trained from scratch, which the paper ties to catastrophic forgetting of its pretraining. A separate test of the mask itself found that attending only to the most recent image scored 7.2% better overall than attending to all previous images.

Freezing the language model is the design and also the limit: its knowledge is frozen too, so the system cannot learn anything that requires changing what the model knows, only how it is conditioned. The connector is also heavy — a resampler plus cross-attention layers threaded through the whole stack is a substantial amount of new machinery to train and to serve.

The interleaved training data was proprietary, which made the result difficult to reproduce and is a large part of why rung 3's much simpler approach was the one that spread.

How it held up: the strategy won and the specific connector did not. Freeze a capable language model, attach a comparatively small bridge from another modality, and train the bridge — that is how essentially every multimodal model is built. And the zero-initialised gate is the single most reusable trick here: adding a component that begins as an exact no-op, so initialisation cannot hurt, appears again in adapter-based fine-tuning on the post-training ladder and is worth reaching for whenever you extend a network you do not want to break.

Flamingo's architecture — freeze strong pretrained encoders, train a lightweight connector — became a dominant template for visual language models. The specific pieces recur throughout the field: resampler-style compression of visual tokens (and its cousins, like Q-Former in BLIP-2), and lightweight connectors that graft vision onto a frozen or lightly-tuned LLM (as in CLIP-plus-projector systems). The zero-initialized gate — adding a new capability by starting from a no-op — shows up wherever you want to extend a pretrained model safely.

Its deeper lesson is about composition. Rather than train one monolithic model to do everything, Flamingo showed you can take excellent specialist models and stitch them together with a small learned interface, inheriting both their strengths at a fraction of the cost. As pretrained components proliferate, that compositional approach — bridge, don't rebuild — has only grown more central to how multimodal systems are built.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Joining text to images, audio, video and finally to a robot arm. Starts with the contrastive objective the whole field standardised on.

1. 1CLIP2021
2. 2Flamingo2022
3. 3LLaVA2023
4. 4wav2vec 2.02020
5. 5Whisper2022
6. 6V-JEPA2024
7. 7RT-22023
8. 8Genie2024

All 110 papers, in 19 ladders →