The big idea

What is multimodal RAG?

Real documents aren't just text — they're charts, diagrams, tables, screenshots, photos and scanned pages. A financial report's key number is in a bar chart; a manual's answer is in a wiring diagram; the insight is in a table's layout. Text-only RAG can't see any of it: OCR and extraction flatten or lose the visual information, so the answer simply isn't in the text it retrieved.

Multimodal RAG retrieves and reasons over multiple modalities. It parses documents into multimodal chunks, embeds them into a shared space so a query can find visual content by meaning, and hands the retrieved images + text to a vision-language model (VLM) that actually sees them and answers, grounded and cited.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end disable multimodal embeddings and blind the generator to see what makes it actually multimodal. Hit Begin.

Step 1 · Text-only loses the picture

Why plain RAG isn't enough

The obvious extension of RAG to a PDF with figures: OCR/extract the text, chunk and embed it, retrieve as usual. What critical information does this throw away?

Design decision: Text-only RAG over OCR'd documents. What does it lose?

The call: Only the fonts. — Formatting is trivial. The real loss is semantic: information encoded visually (charts, diagrams, images, table structure) that has no faithful text representation is simply gone.

Text extraction/OCR discards or flattens everything visual: a chart's trend, a diagram's spatial relationships, a photo's content, and a table's structure (which rows/columns relate). If the answer lives in a figure, text-only RAG never retrieved it because there was no faithful text to embed. To answer questions about visual content, retrieval and generation must both handle the modalities themselves, not a lossy text shadow of them.

Modality is information: Charts, images, diagrams and table layout encode meaning that has no complete text form. Treating documents as text-only throws that away. Multimodal RAG keeps the visual content as a first-class thing to retrieve and to reason over.

Step 2 · Parse into modalities

Multimodal ingestion

Before you can retrieve visual content, you have to extract it as distinct, typed pieces. What does ingestion produce from a rich document?

Parse each document into typed multimodal chunks: text blocks, images/figures, tables, and charts — preserving layout and cross-references (which caption goes with which figure, which page). Use document-layout parsing (not just raw OCR) so a table stays a table and a figure stays a figure with its context. Each chunk keeps a reference to its source asset (the actual image) and location. Now you have a set of pieces in different modalities, ready to embed and index.

Layout-aware parsing: Good ingestion is layout-aware: it segments a page into its structural elements and keeps their relationships, rather than dumping a flat text stream. This preserves tables as tables and figures with their captions — the difference between usable multimodal chunks and mush.

Step 3 · One space for all modalities

Embedding: unified vs convert-to-text

To retrieve a chart with a text query, the chart and the query must be comparable. There are two schools of thought for how to embed mixed modalities. What are they?

Design decision: How do you make an image and a text query comparable for retrieval?

The call: Either unified multimodal embeddings (CLIP-style: image and text in one shared space) or convert-to-text (caption images / extract tables, then embed the text). — Approach one uses a multimodal embedding model that maps images and text into the SAME vector space, so a text query directly retrieves relevant images by meaning (true cross-modal). Approach two generates text descriptions (image captions, table-to-text) and does ordinary text RAG over them. Unified preserves more visual nuance; convert-to-text is simpler and reuses text infra. Many systems combine both.

Two approaches (often combined). Unified multimodal embeddings: a model (CLIP-style) maps images and text into one shared space, so a text query directly retrieves relevant images by meaning — true cross-modal search. Convert-to-text: generate descriptions (caption images, linearize tables) with a VLM and run ordinary text RAG over them. Unified preserves visual nuance and needs no lossy captioning; convert-to-text is simpler and reuses text infrastructure but loses detail. Real systems frequently do both and fuse.

Shared space or shared text: Cross-modal retrieval needs a common ground: either a shared embedding space (image and text vectors comparable) or a shared text representation (everything described as text). The first captures more of the image; the second is easier to build. The choice (or blend) shapes retrieval quality and cost.

Step 4 · Store the vectors

Multimodal index

With chunks embedded, where do the vectors live, and what must you record so retrieval and generation work across modalities?

Store the embeddings in a vector store, tagging each with its modality (text/image/table), a reference to the source asset, and location metadata. Whether unified or convert-to-text, the vectors share a comparable space so a single nearest-neighbor search spans modalities. The critical extra vs text RAG: keep a durable link from each vector to the actual image/asset, because generation will need the real pixels, not just the embedding.

Vectors + asset links: A multimodal index is a vector store plus the bookkeeping to get back the original asset. You retrieve by vector, then fetch the real image for the VLM. Modality tags let you filter or balance results (e.g. ensure a mix of text and figures).

Step 5 · Retrieve across modalities

Cross-modal retrieval

A query arrives — maybe text ("show the revenue trend"), maybe an image ("find diagrams like this"). How do you find the right multimodal chunks?

Embed the query into the same shared space and run nearest-neighbor search to retrieve the most relevant chunks regardless of modality — a text query can surface a chart, an image query can surface similar figures or the text that describes them. Rerank and balance the results (you often want the key figure and its surrounding text), then fetch the actual assets for the ones you'll show the model. The retriever returns a connected multimodal context: the relevant images, tables and text passages.

One query, any modality: Because everything lives in a comparable space, retrieval is modality-agnostic: the query finds the most relevant content whether it's a paragraph, a chart, or a table. Balancing modalities (not just returning all text) and pulling supporting context around a figure improves the answer.

Step 6 · Answer from what it sees

Generation with a vision-language model

Retrieval found the right chart and table. Now the model must read the value off the chart and reason about it. A text-only LLM can't — it never sees the pixels. What generates the answer?

Design decision: Retrieval found the right chart. What must generate the answer, and why?

The call: A text-only LLM using image captions. — Captions are lossy — they can't convey every value in a chart or every relation in a diagram. For real visual reasoning the model must see the actual pixels, i.e. a VLM. (Captions help retrieval and as a fallback, but not for precise visual answers.)

Generate with a vision-language model (VLM): hand it the actual retrieved images (the real pixels) alongside the text, and prompt it to answer from what it sees — reading values off charts, interpreting diagrams, describing photos — with citations to the source figures/passages. Only a model that truly sees the content can answer precise visual questions; a text-only model given captions is working from a lossy shadow. The VLM turns retrieved visuals into a grounded answer.

The generator must see: The final, defining requirement: the answer model receives and reasons over the pixels, not a text summary of them. That's what makes it multimodal RAG rather than text RAG with extra steps. Grounding and citations still apply — answer from the retrieved visuals, cite them.

Step 7 · Assets & delivery

Storing and serving the originals

Unlike text RAG, you're moving images around — into the index pipeline and into the model's context. Where do the originals live and how do they reach the VLM?

Keep the original images/assets in durable object storage, referenced from each vector. At query time, retrieval fetches the actual assets for the top chunks and passes them (as image inputs) to the VLM. Manage size/cost — images are large and VLM image tokens are expensive — by storing sensible resolutions, cropping to the relevant figure, and limiting how many images go into a single generation. Cache popular assets and (as in any media system) serve via CDN where users view them.

Move pixels deliberately: Images are heavy and VLM vision tokens are costly, so you're deliberate: store originals durably, pass only the relevant, right-sized images to the model, and cap the count per answer. The asset store + references are the plumbing that gets real pixels from index to model.

Step 8 · The sharp edges

Modality gap, tables, cost & eval

Multimodal RAG has hard corners: the modality gap (image and text vectors not aligning perfectly), tables/charts that are neither pure image nor clean text, VLM cost for vision, and evaluation that's harder than text.

Mitigate the modality gap (retrieval biasing toward one modality) with good multimodal models, hybrid unified+convert-to-text retrieval, and modality balancing. Handle tables/charts specially — extract structured data and keep the image, since some questions need the numbers and some need the visual. Control cost: vision tokens are pricey, so retrieve tightly, right-size images, and use a VLM only when a visual chunk is actually needed. And evaluate multimodally — check whether answers are truly grounded in the retrieved visuals, not hallucinated, which is harder to measure than text grounding.

Design for the unhappy path: Modality gap → better models + hybrid + balancing. Tables/charts → keep both structure and image. VLM cost → tight retrieval, right-sized images, use vision only when needed. Eval → measure visual grounding. The visual world is richer and messier than text, so multimodal RAG trades more cost and complexity for answers text simply can't give.

You did it

You just designed a multimodal RAG system.

• T — e
• P — a
• M — a
• S — t
• R — e
• G — e
• S — t