LLaVA,
explained.

The multimodal problem is the same as Flamingo's: you have a strong vision encoder (CLIP) and a strong language model, and you want them to work together. Flamingo's answer was architecturally involved — compress vision with a resampler, inject it through new gated cross-attention layers. LLaVA's bet was that a language model, being a general sequence processor, doesn't need special machinery to read images. It just needs the images turned into tokens it can attend to like any other.

So LLaVA does the minimal thing: a single linear projection from image features to the language model's embedding space, and then treats the projected features as ordinary tokens at the start of the prompt. No new attention layers, no resampler. The language model's own self-attention fuses vision and text. It's the kind of simplification that seems too easy to work — and the paper's contribution is showing that, with the right training data, it does.

Concretely: the vision encoder produces feature vectors for an image (one per patch). A trainable projection matrix W maps each feature into the language model's word-embedding dimension, turning it into an "image token." Those image tokens are then simply placed in front of the text tokens, and the whole sequence goes into the language model:

The connector is one matrix — a few million parameters, a tiny fraction (well under 0.1%) of the billions in the language model. Because the image tokens live in the same space as word embeddings and sit in the same sequence, the LLM attends across text and vision with its ordinary self-attention; nothing else changes. The runnable version below shows the projection mapping CLIP's 1024-dim features to a 4096-dim LLM space and the tiny parameter cost. (Later LLaVA versions swap the single linear layer for a small MLP, but the idea is identical.)

A connector alone gives a model that can see but not converse about what it sees. To teach visual instruction-following, you need instruction data — and high-quality multimodal instruction data was scarce. LLaVA's clever move was to synthesize it with a text-only GPT-4. It took images that already had text annotations (captions and object bounding boxes), fed those text descriptions (not the pixels) to GPT-4, and asked it to produce rich instruction-following examples:

The elegance is using a language model's text understanding to bootstrap a vision model's instruction-following, entirely from existing annotations — no human labeling of the multimodal examples. That "generate instruction data with a strong LLM" pattern, which LLaVA brought to the multimodal setting, is now a standard way to build capable assistants cheaply.

Training has two stages, cheap by design. Stage 1 — feature alignment: freeze both the vision encoder and the language model, and train only the projection on image-caption pairs, so the projected image tokens land in a place the LLM understands. Since only the small connector learns, this is fast. Stage 2 — instruction tuning: keep the vision encoder frozen but now fine-tune the projection and the language model together on the GPT-4-generated instruction data, teaching the assistant to follow visual instructions.

The split is efficient: stage 1 does the easy alignment with almost no trainable parameters, and stage 2 spends the real training on teaching behavior. The vision encoder stays frozen throughout — its features are already good. The whole thing was trainable in about a day on modest hardware, which is a large part of why LLaVA was so widely reproduced and extended.

LLaVA punched far above its simplicity:

The result reframed expectations: you didn't need Flamingo-scale machinery to get a good visual assistant. A capable open LLM, CLIP features, a projection, and instruction data were enough — which put strong multimodal models within reach of far more teams.

LLaVA's "projection + prepend + instruction-tune" recipe became the default template for open visual language models. Its two ideas each had outsized influence: the minimal connector (a projection or small MLP, not heavy cross-attention) is now the common way to graft vision onto an LLM, and synthetic instruction data from a strong model is a standard tool for teaching new capabilities cheaply.

The deeper lesson is about where the intelligence already is. A pretrained LLM is a remarkably general processor; LLaVA showed you often don't need to build elaborate new machinery to give it a new modality — you mostly need to speak to it in its own token language and show it examples of the behavior you want. That "align cheaply, then instruction-tune" pattern, and the bias toward minimal connectors, shaped how the field builds multimodal systems today.