LLaVA,
explained.

Rung 2's bridge is a resampler and a set of gated cross-attention layers. This rung's change is to ask how much of that is necessary, and the answer is: almost none of it. Take the vision encoder's output patch features, pass them through a single linear projection into the language model's token embedding space, and prepend them to the text tokens. The image becomes, literally, some tokens at the front of the prompt. No cross-attention, no resampler, no gating — the language model's ordinary self-attention reads the visual tokens because they are in the sequence.

The second half is the data, and it is as economical as the architecture. Visual instruction data — conversations about images — did not exist at scale, so it is generated by a strong text-only model that is shown an image's captions and object boxes, never the image, and asked to write questions, answers and descriptions. Textual descriptions stand in for sight on the teacher's side, and the student learns to do it from pixels. That is the bootstrapping recipe from the post-training ladder, pointed at a modality.

Training is two short stages — align the projection first with everything else frozen, then instruction-tune. Stated against rung 2's units: far less machinery, a training run within reach of an academic budget, and competitive conversational ability about images. This is the rung where a capable vision-language model stopped requiring a frontier lab.

The multimodal problem is the same as Flamingo's: you have a strong vision encoder (CLIP) and a strong language model, and you want them to work together. Flamingo's answer was architecturally involved — compress vision with a resampler, inject it through new gated cross-attention layers. LLaVA's bet was that a language model, being a general sequence processor, doesn't need special machinery to read images. It just needs the images turned into tokens it can attend to like any other.

So LLaVA does the minimal thing: a single linear projection from image features to the language model's embedding space, and then treats the projected features as ordinary tokens at the start of the prompt. No new attention layers, no resampler. The language model's own self-attention fuses vision and text. It's the kind of simplification that seems too easy to work — and the paper's contribution is showing that, with the right training data, it does.

Concretely: the vision encoder produces feature vectors for an image (one per patch). A trainable projection matrix W maps each feature into the language model's word-embedding dimension, turning it into an "image token." Those image tokens are then simply placed in front of the text tokens, and the whole sequence goes into the language model:

The connector is one matrix — a few million parameters, a tiny fraction (well under 0.1%) of the billions in the language model. Because the image tokens live in the same space as word embeddings and sit in the same sequence, the LLM attends across text and vision with its ordinary self-attention; nothing else changes. The runnable version below shows the projection mapping CLIP's 1024-dim features to a 4096-dim LLM space and the tiny parameter cost. (Later LLaVA versions swap the single linear layer for a small MLP, but the idea is identical.)

A connector alone gives a model that can see but not converse about what it sees. To teach visual instruction-following, you need instruction data — and high-quality multimodal instruction data was scarce. LLaVA's clever move was to synthesize it with a text-only GPT-4. It took images that already had text annotations (captions and object bounding boxes), fed those text descriptions (not the pixels) to GPT-4, and asked it to produce rich instruction-following examples:

The elegance is using a language model's text understanding to bootstrap a vision model's instruction-following, entirely from existing annotations — no human labeling of the multimodal examples. That "generate instruction data with a strong LLM" pattern, which LLaVA brought to the multimodal setting, is now a standard way to build capable assistants cheaply.

Training has two stages, cheap by design. Stage 1 — feature alignment: freeze both the vision encoder and the language model, and train only the projection on image-caption pairs, so the projected image tokens land in a place the LLM understands. Since only the small connector learns, this is fast. Stage 2 — instruction tuning: keep the vision encoder frozen but now fine-tune the projection and the language model together on the GPT-4-generated instruction data, teaching the assistant to follow visual instructions.

The split is efficient: stage 1 does the easy alignment with almost no trainable parameters, and stage 2 spends the real training on teaching behavior. The vision encoder stays frozen throughout — its features are already good. The whole thing was trainable in about a day on modest hardware, which is a large part of why LLaVA was so widely reproduced and extended.

LLaVA joins a frozen CLIP image encoder to Vicuna, an open chat model, through one trainable matrix. The architecture fits in a sentence; the method is in the data recipe and the two training stages.

1. Encode the image. CLIP ViT-L/14 at 224 pixels cuts the image into a 16 × 16 grid of 14-pixel patches and returns one 1,024-dimensional feature per patch: 256 vectors. LLaVA takes them from the layer before CLIP's last Transformer layer, which scored better than the last layer in the paper's ablation.
2. Project. A trainable matrix W maps each feature into the language model's word-embedding space, 5,120 dimensions for Vicuna-13B, the main model: about 5 million weights. The 256 results are the image tokens.
3. Format a conversation. A system message, then alternating human and assistant turns. The image tokens go either before or after the first question, chosen at random, and every assistant turn ends with a stop sequence. The loss is computed only on the assistant's answers and stop sequences.
4. Write the instruction data. For COCO images, the text-only GPT-4 was given two symbolic descriptions: the human-written captions, and the object bounding boxes with their categories. A few hand-written seed examples per data type, the only human annotation in the pipeline, served as in-context examples. The result, LLaVA-Instruct-158K, holds 58K multi-turn conversations, 23K detailed descriptions and 77K complex-reasoning answers. In early tests GPT-4 wrote better data than ChatGPT, notably for spatial reasoning.
5. Stage 1: align the features. CC3M was filtered to 595K image–caption pairs. Each becomes a one-turn exchange: a randomly chosen request for a brief description, answered by the original caption. CLIP and Vicuna stay frozen and only W trains, for one epoch at learning rate 2×10^⁻³ with batch size 128, in under 4 hours on 8 A100 GPUs.
6. Stage 2: instruction-tune. CLIP stays frozen; W and all of Vicuna train on the 158K examples for three epochs at learning rate 2×10^⁻⁵ with batch size 32, in under 10 hours on the same 8 GPUs. A separate model is fine-tuned on ScienceQA, generating its reasoning before its answer.
7. Answer. At inference the model reads the image tokens and the question and generates its reply like any chat model.

What LLaVA-1.5 and LLaVA-NeXT changed

LLaVA-1.5, a follow-up paper by the same group, replaced W with a two-layer MLP, moved to CLIP at 336 pixels (a 24 × 24 grid, so 576 tokens), switched to Vicuna v1.5, and mixed academic VQA datasets into a 665K-example instruction set, using format prompts such as "Answer the question using a single word or phrase." for short answers. The whole run took about a day on one node of 8 A100 GPUs. LLaVA-NeXT later split higher-resolution images into several 336-pixel tiles plus a downscaled overview, encoding each, which multiplies the tokens per image.

• Xv — the image; g — the frozen CLIP ViT-L/14, read at its second-to-last layer; Zv — 256 features of 1,024 dimensions
• W — the trainable projection; Hv — 256 vectors in the language model's embedding space
• Xinstruct — the human turns; Xa — the assistant's answers; xi — the i-th answer token, so the product and its loss cover answer tokens only
• φ — Vicuna's own weights

Training minimizes −log of that probability. At the extremes: with W = 0 every image token is the zero vector and the model becomes a text-only chatbot that sees 256 blank positions. Skipping stage 1 means the later fine-tune starts from a random W, and on ScienceQA that cost about 5 points (90.92% down to 85.81%). And if the loss also covered the human turns, the model would learn to write the questions as well, spending gradient on text the user always supplies.

LLaVA was measured two ways. LLaVA-Bench uses GPT-4 as a judge: a text-only GPT-4 first writes a reference answer from a textual description of the image, then rates the reference and LLaVA's answer from 1 to 10 for helpfulness, relevance, accuracy and detail, and LLaVA's score is reported relative to the reference. The COCO version has 30 images and 90 questions, three types per image; the In-the-Wild version has 24 more varied images and 60 questions. ScienceQA, a multiple-choice science benchmark, is scored by plain accuracy.

The last row combines two models: when LLaVA and GPT-4 disagree, GPT-4 is asked again with both answers and gives a final one, which set a new ScienceQA record. The ScienceQA ablations isolate the design choices: skipping stage 1 gave 85.81%, reading CLIP's last layer instead of the one before it 89.96%, and the 7B model instead of 13B 89.84%.

The paper also shows the family's characteristic failure. Asked whether a fridge held strawberry-flavored yogurt, LLaVA said yes when the photo showed yogurt and, separately, strawberries; the authors read this as the model sometimes treating an image as a bag of patches.

The cost of prepending rather than resampling is context: every visual token occupies a position in the sequence, so an image costs hundreds of tokens, and higher resolution or several images multiply that directly. Rung 2's fixed-size compression was solving a real problem, and this rung gives that up in exchange for simplicity — which is why the later work in this family is largely about tiling high-resolution images and then compressing the resulting tokens again.

The synthetic data has the failure mode its construction predicts. A teacher writing about an image it cannot see, from captions and boxes, will occasionally describe things that are not there — so the student is trained on some confidently wrong visual assertions, and models of this family are known for fluent descriptions of absent details. Evaluation also leaned on model-as-judge comparisons, which carry documented biases toward length and style.

How it held up: projection-plus-prepend is the dominant architecture. Nearly every open vision-language model is this shape, with a better vision encoder, a better language model and more data. The lesson is one the site keeps finding: the elaborate connector was solving a problem that mostly dissolved once the language model was strong enough to handle visual tokens in its ordinary input, and the simplest bridge that fits in the existing interface won.

LLaVA's "projection + prepend + instruction-tune" recipe became the default template for open visual language models. Its two ideas each had outsized influence: the minimal connector (a projection or small MLP, not heavy cross-attention) is now the common way to graft vision onto an LLM, and synthetic instruction data from a strong model is a standard tool for teaching new capabilities cheaply.

The deeper lesson is about where the intelligence already is. A pretrained LLM is a remarkably general processor; LLaVA showed you often don't need to build elaborate new machinery to give it a new modality — you mostly need to speak to it in its own token language and show it examples of the behavior you want. That "align cheaply, then instruction-tune" pattern, and the bias toward minimal connectors, shaped how the field builds multimodal systems today.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Joining text to images, audio, video and finally to a robot arm. Starts with the contrastive objective the whole field standardised on.

1. 1CLIP2021
2. 2Flamingo2022
3. 3LLaVA2023
4. 4wav2vec 2.02020
5. 5Whisper2022
6. 6V-JEPA2024
7. 7RT-22023
8. 8Genie2024

All 110 papers, in 19 ladders →