The whole design, in writing
Learn AI system design by building a retrieval-augmented generation (RAG) pipeline step by step. An interactive guide covering document chunking, embeddings, the vector store, top-k retrieval, reranking, grounded prompt assembly, and keeping the index fresh — so an LLM answers from your knowledge, not its training data.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Why retrieval at all?
An LLM only knows what it was trained on. Ask it about your internal docs, last week’s release, or a private wiki and it confidently makes something up. How do you make it answer from knowledge it never saw?
Retrieval-Augmented Generation: before generating, fetch the most relevant passages from your knowledge and put them in the prompt. The model reasons over real, current text instead of fuzzy memory — and can cite its sources.
What the new pieces do
- Userclient
- A person asking a question that should be answered from your documents, not the model’s training data.
Step 1 · The skeleton
Retrieve, then generate
A user asks a question. We can’t just forward it to the LLM — it would answer from training data. What has to happen between the question and the answer?
The user asks a question about your private docs. What’s the flow?
It has never seen your documents, so it answers from training data — confidently and often wrong. This is exactly what RAG exists to fix.
A gateway orchestrates retrieve-then-generate: find the chunks that matter, hand them to the model as context, and let it answer from real text.
Fine-tuning bakes knowledge in slowly and expensively, can’t cite sources, and goes stale the moment a doc changes. Retrieval keeps knowledge live and external.
A RAG Gateway orchestrates two phases for every question: retrieve the most relevant passages, then generate an answer grounded in them. Knowledge lives outside the model, so it’s always current and citable.
What the new pieces do
- RAG Gatewaybackend
- Coordinates the retrieve-then-generate flow: embeds the query, fetches chunks, builds the grounded prompt, calls the model.
Step 2 · Prepare the knowledge
Chunk and embed your documents
Your knowledge is a pile of long documents. You can’t hand a whole 80-page PDF to the model per query. How do you make documents searchable by meaning?
How do you store documents so you can find the relevant bits by meaning?
Keyword search misses paraphrase — "how do I cancel" won’t match "terminating your subscription." Meaning needs semantic vectors, not exact words.
Whole docs are too big for the context window and too coarse to pinpoint the relevant passage. You need to split first.
An offline pipeline chunks documents, embeds each chunk into a vector capturing its meaning, and upserts vectors + text into the stores. Now "find by meaning" is a nearest-neighbour search.
An offline Ingestion pipeline loads each document, splits it into chunks, runs every chunk through an Embedding Model to get a vector, and upserts the vectors into a Vector Store (plus the raw text into a Document Store). This is the prep work that makes retrieval possible.
- ~300–800tokens / chunk
- 1 vectorper chunk
- offlineruns ahead of time
What the new pieces do
- Vector Storeindex
- Holds chunk embeddings and answers nearest-neighbour search in milliseconds via an ANN index.
- Document Storestore
- The original chunk text and metadata, returned alongside vectors so the prompt and citations use real content.
- Ingestionbus
- The offline pipeline: load documents, split into chunks, embed each, and upsert into the vector + document stores.
- Knowledgestore
- The source of truth: PDFs, docs, wikis, tickets — whatever the system must answer from.
- Embedding Modelbus
- Maps text to vectors. The SAME model must embed both stored chunks and incoming queries so they share a space.
Back of the envelope
- split on structure
- paragraphs/sections, not arbitrary character counts
- add overlap
- ~10–15% so ideas spanning a boundary aren’t cut in half
- same embedding model
- for chunks AND queries — they must share a space
- store metadata
- source, title, URL — for filtering and citations
Step 3 · Find the right chunks
Embed the query, search by similarity
A question comes in. The chunks are vectors in a store. How do you find the handful that actually answer this question — out of possibly millions?
You have millions of chunk vectors. How do you find the closest to the query?
Exact nearest-neighbour over millions of high-dim vectors is too slow per query. At scale you trade a little accuracy for huge speed.
Embed the query with the SAME model, then let an ANN index (HNSW/IVF) return the closest chunks in milliseconds — near-exact recall, sub-linear cost.
That’s back to lexical search and its paraphrase blind spot. Semantic retrieval is the whole point. (Hybrid keyword+vector is a refinement, not a replacement.)
The Retriever embeds the query with the same model, then asks the Vector Store for the top-k nearest chunks via an ANN index. The Document Store returns their text. Semantic match means "cancel my plan" finds "terminating your subscription."
What the new pieces do
- Retrieverservice
- Turns the query into a vector and searches the store for the nearest chunks (semantic, not keyword, match).
Step 4 · How much to fetch
Top-k and the recall/precision dial
Retrieve too few chunks and you miss the answer. Retrieve too many and you bury the model in noise, blow the token budget, and slow it down. Where’s the line?
How many chunks should you stuff into the prompt?
More context isn’t better — irrelevant chunks distract the model ("lost in the middle"), raise cost, and slow generation. Precision beats volume.
Too brittle: the answer often spans two or three passages, and the very top hit isn’t always the right one. You need a small set, not a single bet.
Retrieve a modest candidate set (e.g. k≈20), then narrow to the best few. Enough recall to catch the answer, enough precision to keep the prompt clean.
Retrieve a modest top-k candidate set from the Vector + Document stores — wide enough that the answer is almost certainly in there (recall), but not so wide it drowns the prompt. The next step tightens it to the best few (precision).
Back of the envelope
- k ≈ 20 candidates × ~500 tok/chunk (mid-range of Step 2’s ~300–800) ≈ 10,000 tok
- the retrieval-stage set — only the reranker reads this, not the generator
- → rerank → top 3–5 × ~500 tok ≈ 1,500–2,500 tok
- what actually lands in the prompt
- on an 8K-context model, worst case that leaves ≈ 5,500+ tok
- room for the system prompt, conversation history, and the answer
- optional metadata filter
- restrict by source/recency before ranking
Step 5 · Sharpen the shortlist
Rerank for precision
The vector search is fast but coarse — it ranks by embedding similarity, which isn’t the same as "actually answers the question." The true best chunk might sit at position 8. How do you fix the order?
ANN similarity ≠ true relevance. How do you get the best chunks to the top?
Bi-encoder similarity is a fast approximation. The genuinely most relevant passage often isn’t the nearest vector — order needs a second, sharper pass.
A cross-encoder reads the query and each chunk together and scores true relevance — far more accurate. Run it on the ~20 candidates (cheap), keep the best 3–5.
More candidates raises recall but not precision — you still feed the model noise. Reranking is what turns a long candidate list into a clean shortlist.
A Reranker (a cross-encoder) re-scores the top-k candidates by reading the query and each chunk together, then keeps only the best 3–5. It’s too slow to run over the whole store — but perfect over a 20-candidate shortlist.
What the new pieces do
- Rerankerservice
- Re-scores the top candidates with a heavier cross-encoder so the few chunks that reach the prompt are the most relevant.
Step 6 · Ground the answer
Build the prompt and generate
You have the 3–5 best chunks. Now the model has to answer — but you need it to use those chunks and not slip back into making things up. How do you assemble the prompt?
You have the best chunks. How do you get a grounded, citable answer?
If the chunks aren’t in the prompt, the model can’t use them. Grounding only works when the retrieved text is actually in the context.
The gateway builds a grounded prompt: "answer ONLY from these passages, cite them, say you don’t know if they don’t cover it" — then generates.
Back to noise and cost. You reranked for a reason — feed the clean shortlist, not the raw candidate pile.
The Gateway assembles a grounded prompt — system instructions + the reranked chunks + the question — and tells the LLM to answer only from the passages, cite them, and admit when they don’t cover the question. The chunks carry their source metadata, so citations point at real documents.
What the new pieces do
- LLMservice
- Generates the answer using only the retrieved chunks as context, with instructions to cite and not invent.
Step 7 · Trust the answer
Citations, "I don’t know", and evals
Even grounded, the model can over-claim or cite the wrong chunk. Users need to trust the answer — and you need to know when retrieval is failing. How do you keep RAG honest?
How do you keep a RAG system trustworthy over time?
Grounding reduces hallucination but doesn’t eliminate it — the model can still misread or over-generalize from a chunk. Trust must be verifiable.
Citations are the trust mechanism — they let a user verify the claim against the source. Hiding them removes the one thing that makes RAG auditable.
Surface the source chunks as citations, let the model decline when context is thin, and run evals (retrieval recall, answer faithfulness) to catch drift.
The answer ships with citations back to the source chunks, the model is allowed to say "I don’t know" when retrieval comes up thin, and an eval harness tracks retrieval quality (did we fetch the right chunks?) and answer faithfulness (did the answer stick to them?). That’s how you catch a silently drifting index before users do.
The payoff
You built a RAG pipeline
From a forgetful model to a grounded, citable system: an ingestion pipeline that chunks and embeds, a vector store for semantic search, two-stage retrieval with reranking, grounded generation, and evals to keep it honest.
Now stale the index and watch RAG’s signature failure — confident answers built on outdated chunks — and see why ingestion is a first-class, ongoing system, not a one-time script.
Everything you assembled, in order
- RAG Gateway — orchestrates retrieve → augment → generate
- Ingestion — chunk + embed documents, offline and ongoing
- Embeddings — meaning as coordinates — same model for chunks + queries
- Vector Store — ANN search returns top-k by similarity in ms
- Top-k — wide for recall, then narrowed for precision
- Reranker — cross-encoder sharpens the shortlist to the best few
- Grounded prompt — answer only from context, cite, decline if thin
- Evals + citations — make a quiet failure mode visible and auditable