The whole design, in writing
Learn AI system design by building a multimodal RAG system that answers questions grounded in images, charts, tables and PDFs step by step. An interactive guide covering why text-only RAG loses visual information, parsing documents into multimodal chunks, unified multimodal embeddings vs convert-to-text, cross-modal retrieval, generation with a vision-language model that actually sees the content, asset storage, and the unhappy paths (modality gap, tables/charts, VLM cost, evaluation).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is multimodal RAG?
Real documents aren't just text — they're charts, diagrams, tables, screenshots, photos and scanned pages. A financial report's key number is in a bar chart; a manual's answer is in a wiring diagram; the insight is in a table's layout. Text-only RAG can't see any of it: OCR and extraction flatten or lose the visual information, so the answer simply isn't in the text it retrieved.
Multimodal RAG retrieves and reasons over multiple modalities. It parses documents into multimodal chunks, embeds them into a shared space so a query can find visual content by meaning, and hands the retrieved images + text to a vision-language model (VLM) that actually sees them and answers, grounded and cited.
What the new pieces do
- Clientclient
- Asks a question whose answer may live in a figure, chart, table or photo — or even queries with an image.
Step 1 · Text-only loses the picture
Why plain RAG isn't enough
The obvious extension of RAG to a PDF with figures: OCR/extract the text, chunk and embed it, retrieve as usual. What critical information does this throw away?
Text-only RAG over OCR'd documents. What does it lose?
A bar chart's trend, a diagram's spatial relationships, a photo's content, and a table's row/column structure carry meaning that OCR either loses entirely (no text) or flattens into a jumble. If the answer is "in the picture", text-only retrieval never had it to retrieve.
Formatting is trivial. The real loss is semantic: information encoded visually (charts, diagrams, images, table structure) that has no faithful text representation is simply gone.
It doesn't. Vast amounts of meaning in real documents live in figures, charts, tables and images; flattening to text discards it, so text-only RAG can't answer questions about that content.
Text extraction/OCR discards or flattens everything visual: a chart's trend, a diagram's spatial relationships, a photo's content, and a table's structure (which rows/columns relate). If the answer lives in a figure, text-only RAG never retrieved it because there was no faithful text to embed. To answer questions about visual content, retrieval and generation must both handle the modalities themselves, not a lossy text shadow of them.
Why this piece earns its place
Nobody files this as a bug, and that is the part to plan around. A user whose visual question came back as a plausible paragraph concludes the system does not do charts, and stops asking — so the queries that would have proved the gap drain out of your traffic, and the logs look healthiest exactly where coverage is worst. Demand you never served leaves no trace. So you go at it from the corpus side instead: sample pages at random, count how many carry a chart, a table or a diagram, and hand-write questions that can only be answered from those. That count is also the honest case for skipping all of this — over tickets, chat logs and code comments the visual share is near nothing, and everything that follows is cost for no answers gained. Establish it early, because the decision is expensive to reverse: going multimodal moves parsing, the index and the generator together, and each of those means re-processing the corpus rather than flipping a flag.
Step 2 · Parse into modalities
Multimodal ingestion
Before you can retrieve visual content, you have to extract it as distinct, typed pieces. What does ingestion produce from a rich document?
Parse each document into typed multimodal chunks: text blocks, images/figures, tables, and charts — preserving layout and cross-references (which caption goes with which figure, which page). Use document-layout parsing (not just raw OCR) so a table stays a table and a figure stays a figure with its context. Each chunk keeps a reference to its source asset (the actual image) and location. Now you have a set of pieces in different modalities, ready to embed and index.
Why this piece earns its place
Parsing is the most expensive and most fragile stage here, and it deserves to be treated as its own cached artifact. Keep the parse output keyed by document hash plus parser version, because re-parsing a corpus of scanned PDFs costs far more than re-embedding it — and you will re-parse, since the layout model is the piece you swap most often. That version stamp matters for a second reason: a parser upgrade re-segments the page, so any chunk id derived from position silently becomes a different chunk, and re-ingesting appends a second copy of a figure rather than replacing the first. The quiet breakage to watch for is figure-caption association. A two-column layout hands a figure the neighbouring column's caption, and nothing downstream can detect it — the retrieved image is wrong in a way that is perfectly consistent, and the answer cites it confidently. Where a source is born-digital and already tagged, skip the layout model and read the structure you were given; layout parsing exists to recover structure that was thrown away at export.
- typed chunkstext · image · table
- layout-awarekeep structure
- asset refslink to originals
What the new pieces do
- MM-RAG Svcgateway
- Ingests documents into multimodal chunks, and at query time retrieves across modalities and calls a vision-language model.
Step 3 · One space for all modalities
Embedding: unified vs convert-to-text
To retrieve a chart with a text query, the chart and the query must be comparable. There are two schools of thought for how to embed mixed modalities. What are they?
How do you make an image and a text query comparable for retrieval?
That's text-only RAG again, blind to visuals. You need images to be retrievable, either by embedding them into a shared space with text or by describing them as text first.
Unindexed images can't be found by a query. Images must be made retrievable — via multimodal embeddings or via generated text descriptions.
Approach one uses a multimodal embedding model that maps images and text into the SAME vector space, so a text query directly retrieves relevant images by meaning (true cross-modal). Approach two generates text descriptions (image captions, table-to-text) and does ordinary text RAG over them. Unified preserves more visual nuance; convert-to-text is simpler and reuses text infra. Many systems combine both.
Two approaches (often combined). Unified multimodal embeddings: a model (CLIP-style) maps images and text into one shared space, so a text query directly retrieves relevant images by meaning — true cross-modal search. Convert-to-text: generate descriptions (caption images, linearize tables) with a VLM and run ordinary text RAG over them. Unified preserves visual nuance and needs no lossy captioning; convert-to-text is simpler and reuses text infrastructure but loses detail. Real systems frequently do both and fuse.
Why this piece earns its place
Two operational facts decide this choice more often than fidelity does. Convert-to-text is a model call per image at ingest, so ingestion becomes an inference bill and a backfill becomes a re-captioning run over the whole corpus. Those captions are model output, not source data — store each one with the model that wrote it, or the day retrieval quality shifts you will have no way to tell whether the corpus changed or the captioner did. Second: if you run both paths you have two vector spaces, and their scores are not on the same scale, so fuse by rank rather than by score. Averaging similarity numbers from two different models reads as principled and is arithmetic on incomparable units. One more thing worth saying in an interview — a shared space is a property of models trained together, not of matching dimensions. Bolting an off-the-shelf image encoder next to your text encoder gives you two unrelated spaces of the same width, and nearest-neighbour across them is noise that still returns results.
- unifiedimage+text shared space
- convert-to-textcaption then text RAG
- often bothfuse the two
What the new pieces do
- Doc Parserservice
- Parses PDFs/docs into typed chunks — text blocks, images/figures, tables, charts — preserving layout and references.
Step 4 · Store the vectors
Multimodal index
With chunks embedded, where do the vectors live, and what must you record so retrieval and generation work across modalities?
Store the embeddings in a vector store, tagging each with its modality (text/image/table), a reference to the source asset, and location metadata. Whether unified or convert-to-text, the vectors share a comparable space so a single nearest-neighbor search spans modalities. The critical extra vs text RAG: keep a durable link from each vector to the actual image/asset, because generation will need the real pixels, not just the embedding.
Why this piece earns its place
The link from a vector to its asset crosses two systems with no transaction between them, and that seam is where this design actually breaks. Delete or overwrite an object and the vector survives, still ranked, still returned — and the fetch now fails on the critical path, mid-request, after retrieval has already spent its latency budget. So point at an immutable key, retire assets with a tombstone rather than a delete, and decide in advance what generation does when a fetch fails: drop that chunk and note it, rather than failing an answer the remaining chunks could have carried. The other thing to get right at write time is provenance — not what the chunk is, which the tags already say, but how it came to exist: which parser version cut it, and which box on the page it was cut from. Neither can be recovered from a row afterwards, both are free to write while the row is being written, and together they turn the question that eventually arrives — one parse run was bad, which vectors came out of it — from a re-ingest of the whole corpus into a filter over the index.
- vector storeone comparable space
- modality tagstext/image/table
- asset linkback to pixels
What the new pieces do
- MM Embedderservice
- Embeds each chunk (image or text) into a shared multimodal space so a query can match content in any modality.
- Vector Storedata
- Stores the multimodal embeddings (with modality + source refs) for cross-modal nearest-neighbor search.
Step 5 · Retrieve across modalities
Cross-modal retrieval
A query arrives — maybe text ("show the revenue trend"), maybe an image ("find diagrams like this"). How do you find the right multimodal chunks?
Embed the query into the same shared space and run nearest-neighbor search to retrieve the most relevant chunks regardless of modality — a text query can surface a chart, an image query can surface similar figures or the text that describes them. Rerank and balance the results (you often want the key figure and its surrounding text), then fetch the actual assets for the ones you'll show the model. The retriever returns a connected multimodal context: the relevant images, tables and text passages.
Why this piece earns its place
Two things change once retrieval spans modalities. First, the modality mix belongs in the request, not the config. Show me the revenue trend wants figures; what does the warranty cover wants prose; one fixed quota serves one of them badly. Apply that mix as a quota over the ranked list rather than as a filter ahead of the search — filter first and a visual question comes back as the five best figures with none of the prose that explains them, and you threw away recall you had already paid for. Second, retrieval is no longer just the search. Once the shortlist is settled the assets still have to be pulled, and those object-store round trips sit on the critical path — the nearest-neighbour query was never the slow part here. Issue them concurrently rather than walking the shortlist one chunk at a time, and expect the tail to be set by the largest image in the set, not the average one. And measure recall per modality. An aggregate number stays healthy while figure recall collapses, because prose is the bulk of most corpora — the system quietly reverts to text-only RAG and no dashboard mentions it.
- query → shared spacetext or image
- cross-modaltext finds charts
- fetch assetsfor generation
What the new pieces do
- Retrieverservice
- Query time: embeds the query and finds the most relevant multimodal chunks across text, images, tables and charts.
Step 6 · Answer from what it sees
Generation with a vision-language model
Retrieval found the right chart and table. Now the model must read the value off the chart and reason about it. A text-only LLM can't — it never sees the pixels. What generates the answer?
Retrieval found the right chart. What must generate the answer, and why?
The generator must be a multimodal LLM that is handed the real images (plus text) and can actually see them — reading a value off a chart, interpreting a diagram, describing a photo — to produce an answer grounded in the visual content, with citations to the sources. A text-only model given only captions can't do this.
Captions are lossy — they can't convey every value in a chart or every relation in a diagram. For real visual reasoning the model must see the actual pixels, i.e. a VLM. (Captions help retrieval and as a fallback, but not for precise visual answers.)
Returning the raw image punts the reasoning to the user. The point is for the model to interpret the visual content and answer the question, which requires a vision-capable model.
Generate with a vision-language model (VLM): hand it the actual retrieved images (the real pixels) alongside the text, and prompt it to answer from what it sees — reading values off charts, interpreting diagrams, describing photos — with citations to the source figures/passages. Only a model that truly sees the content can answer precise visual questions; a text-only model given captions is working from a lossy shadow. The VLM turns retrieved visuals into a grounded answer.
Why this piece earns its place
A visual citation is a different object from a textual one. Pointing a user at a page number does not let them check a value read off a chart — they need the exact picture the model was shown, so the response carries that asset reference beside the claim, and it has to still resolve when they click it. This is also the debugging story. A wrong visual answer has two obvious causes — retrieval fetched the wrong figure, or the model misread the right one — and you cannot tell them apart unless you logged which images went in and how they were prepared: the region cut out, and the size it was sent at. A third cause hides behind the second, because a figure shrunk until its axis labels blur produces exactly the symptom of a model that cannot read charts. One reliability note people learn the hard way: the vision path fails differently from the text path. A single oversized or unsupported image rejects the whole call, so validate dimensions and format at ingest, where one document fails, rather than at generation, where one user's request does.
- VLMsees the pixels
- answer from visualsread the chart
- citesource figures
What the new pieces do
- Vision-LLMservice
- A multimodal LLM that receives the actual retrieved images + text and grounds its cited answer in what it sees.
- Asset Storedata
- Durable store of the original images/assets; the VLM is handed the actual pixels, not just a pointer.
Step 7 · Assets & delivery
Storing and serving the originals
Unlike text RAG, you're moving images around — into the index pipeline and into the model's context. Where do the originals live and how do they reach the VLM?
Keep the original images/assets in durable object storage, referenced from each vector. At query time, retrieval fetches the actual assets for the top chunks and passes them (as image inputs) to the VLM. Manage size/cost — images are large and VLM image tokens are expensive — by storing sensible resolutions, cropping to the relevant figure, and limiting how many images go into a single generation. Cache popular assets and (as in any media system) serve via CDN where users view them.
Why this piece earns its place
What you hand the model is never the original — it is a rendition, a crop at a chosen resolution. Treat that set of renditions as its own cache and build it at ingest, or you pay a transcode on the critical path every time a popular figure is retrieved. The catch is that you have to choose sizes before you know what any question will need, so keep the original and treat a miss as something you generate once and then keep. The part that is easy to miss until a security review: these images get served to browsers, so the URL is the access boundary. A citation that links to a public bucket path is a permanent public link into a private document, sitting in a chat transcript forever — issue short-lived signed URLs instead. Retraction has to reach the renditions and not just the page they came from — a crop is not a thumbnail of a document, it is the part of it somebody reads. Retire them the way the original is retired, as a key marked gone rather than a key erased, so a citation written last month resolves to a withdrawn notice instead of a fetch that fails with nothing to say for itself.
- object storedurable originals
- right-sizecrop, resolution
- limitimages per generation
Step 8 · The sharp edges
Modality gap, tables, cost & eval
Multimodal RAG has hard corners: the modality gap (image and text vectors not aligning perfectly), tables/charts that are neither pure image nor clean text, VLM cost for vision, and evaluation that's harder than text.
Mitigate the modality gap (retrieval biasing toward one modality) with good multimodal models, hybrid unified+convert-to-text retrieval, and modality balancing. Handle tables/charts specially — extract structured data and keep the image, since some questions need the numbers and some need the visual. Control cost: vision tokens are pricey, so retrieve tightly, right-size images, and use a VLM only when a visual chunk is actually needed. And evaluate multimodally — check whether answers are truly grounded in the retrieved visuals, not hallucinated, which is harder to measure than text grounding.
Why this piece earns its place
Evaluation is the piece that actually stalls teams, and it is worth naming why. Checking whether an answer is grounded in a chart means the judge has to see the chart too — so your automated judge is itself a vision model, grading output from the same class of model that produced it and sharing its blind spots. Keep a small human-labelled anchor set and score the judge against it periodically; without one, a judge regression looks exactly like a system improvement. The modality gap does something quieter to your thresholds, too. Any cutoff that decides a result is too weak to use gets calibrated on whichever modality dominated the traffic you calibrated against, and it will then reject the other one at a different rate — so set that cutoff per modality, or you will be suppressing figures in the belief that you tuned for quality. And the cost shape is genuinely different from text RAG: per-query cost is bimodal, because a visual answer and a prose answer are not the same product, so an average tells you nothing and capacity planning has to run on the visual share of traffic. That share is not static. It grows as users discover the system can read their charts, which means a budget modelled on launch-week traffic is wrong in the direction that hurts.
You did it
You just designed a multimodal RAG system.
Everything you assembled, in order
- Text-only RAG loses everything visual — charts, diagrams, photos, and table structure that OCR flattens.
- Parse documents into typed multimodal chunks (text, images, tables) with layout and asset references.
- Make modalities comparable via unified multimodal embeddings (shared space) and/or convert-to-text — often both.
- Store vectors with modality tags and durable links back to the original assets.
- Retrieve cross-modally: one query (text or image) finds the relevant chunks in any modality.
- Generate with a vision-language model that receives the actual pixels and answers from what it sees, cited.
- Store/serve originals deliberately (right-size, cap per answer); mind the modality gap, tables, VLM cost and eval.
