The big idea

Why retrieval at all?

An LLM only knows what it was trained on. Ask it about your internal docs, last week’s release, or a private wiki and it confidently makes something up. How do you make it answer from knowledge it never saw?

Retrieval-Augmented Generation: before generating, fetch the most relevant passages from your knowledge and put them in the prompt. The model reasons over real, current text instead of fuzzy memory — and can cite its sources.

How to read this: Each step opens with a real design decision — you make the call before I show you what ships. Watch the diagram grow, hover any box, replay the flow. At the end stale the index to see RAG’s quiet failure mode. Hit Begin.

Step 1 · The skeleton

Retrieve, then generate

A user asks a question. We can’t just forward it to the LLM — it would answer from training data. What has to happen between the question and the answer?

Design decision: The user asks a question about your private docs. What’s the flow?

The call: Retrieve relevant passages first, then generate from them. — A gateway orchestrates retrieve-then-generate: find the chunks that matter, hand them to the model as context, and let it answer from real text.

A RAG Gateway orchestrates two phases for every question: retrieve the most relevant passages, then generate an answer grounded in them. Knowledge lives outside the model, so it’s always current and citable.

Retrieve → augment → generate: The model’s job shrinks to "answer using these passages." Correctness moves from "did the model memorize this?" to "did we retrieve the right context?" — a problem you can actually engineer.

Step 2 · Prepare the knowledge

Chunk and embed your documents

Your knowledge is a pile of long documents. You can’t hand a whole 80-page PDF to the model per query. How do you make documents searchable by meaning?

Design decision: How do you store documents so you can find the relevant bits by meaning?

The call: Split into chunks, embed each into a vector, and store the vectors. — An offline pipeline chunks documents, embeds each chunk into a vector capturing its meaning, and upserts vectors + text into the stores. Now "find by meaning" is a nearest-neighbour search.

An offline Ingestion pipeline loads each document, splits it into chunks, runs every chunk through an Embedding Model to get a vector, and upserts the vectors into a Vector Store (plus the raw text into a Document Store). This is the prep work that makes retrieval possible.

Embeddings = meaning as coordinates: An embedding maps text to a point in high-dimensional space where similar meanings sit close together. Chunking controls granularity; the embedding model controls what "similar" means.

Step 3 · Find the right chunks

Embed the query, search by similarity

A question comes in. The chunks are vectors in a store. How do you find the handful that actually answer this question — out of possibly millions?

Design decision: You have millions of chunk vectors. How do you find the closest to the query?

The call: Embed the query and run approximate nearest-neighbour (ANN) search. — Embed the query with the SAME model, then let an ANN index (HNSW/IVF) return the closest chunks in milliseconds — near-exact recall, sub-linear cost.

The Retriever embeds the query with the same model, then asks the Vector Store for the top-k nearest chunks via an ANN index. The Document Store returns their text. Semantic match means "cancel my plan" finds "terminating your subscription."

Approximate, on purpose: Exact search over millions of vectors is too slow, so ANN indexes (HNSW, IVF) trade a sliver of recall for orders-of-magnitude speed. You tune k and the index to balance recall vs latency.

Step 4 · How much to fetch

Top-k and the recall/precision dial

Retrieve too few chunks and you miss the answer. Retrieve too many and you bury the model in noise, blow the token budget, and slow it down. Where’s the line?

Design decision: How many chunks should you stuff into the prompt?

The call: A small top-k, then tighten precision before generating. — Retrieve a modest candidate set (e.g. k≈20), then narrow to the best few. Enough recall to catch the answer, enough precision to keep the prompt clean.

Retrieve a modest top-k candidate set from the Vector + Document stores — wide enough that the answer is almost certainly in there (recall), but not so wide it drowns the prompt. The next step tightens it to the best few (precision).

Recall vs precision: First-stage retrieval optimizes recall (don’t miss the answer); the next stage optimizes precision (only the best reach the model). Splitting the job is why two-stage retrieval beats one.

Step 5 · Sharpen the shortlist

Rerank for precision

The vector search is fast but coarse — it ranks by embedding similarity, which isn’t the same as "actually answers the question." The true best chunk might sit at position 8. How do you fix the order?

Design decision: ANN similarity ≠ true relevance. How do you get the best chunks to the top?

The call: Rerank the candidates with a cross-encoder, then keep the top few. — A cross-encoder reads the query and each chunk together and scores true relevance — far more accurate. Run it on the ~20 candidates (cheap), keep the best 3–5.

A Reranker (a cross-encoder) re-scores the top-k candidates by reading the query and each chunk together, then keeps only the best 3–5. It’s too slow to run over the whole store — but perfect over a 20-candidate shortlist.

Two-stage retrieval: Stage 1 (vectors) is cheap and wide for recall. Stage 2 (reranker) is expensive and narrow for precision. The pattern — fast filter, then precise rank — is everywhere in search.

Step 6 · Ground the answer

Build the prompt and generate

You have the 3–5 best chunks. Now the model has to answer — but you need it to use those chunks and not slip back into making things up. How do you assemble the prompt?

Design decision: You have the best chunks. How do you get a grounded, citable answer?

The call: Assemble system instructions + chunks + question, and instruct it to cite and not invent. — The gateway builds a grounded prompt: "answer ONLY from these passages, cite them, say you don’t know if they don’t cover it" — then generates.

The Gateway assembles a grounded prompt — system instructions + the reranked chunks + the question — and tells the LLM to answer only from the passages, cite them, and admit when they don’t cover the question. The chunks carry their source metadata, so citations point at real documents.

Grounding + guardrails: Two instructions do the heavy lifting: "use only the provided context" (stops invention) and "if it’s not here, say so" (stops confident wrong answers). Retrieval supplies the facts; the prompt supplies the discipline.

Step 7 · Trust the answer

Citations, "I don’t know", and evals

Even grounded, the model can over-claim or cite the wrong chunk. Users need to trust the answer — and you need to know when retrieval is failing. How do you keep RAG honest?

Design decision: How do you keep a RAG system trustworthy over time?

The call: Return citations, allow "I don’t know," and evaluate retrieval + answers. — Surface the source chunks as citations, let the model decline when context is thin, and run evals (retrieval recall, answer faithfulness) to catch drift.

The answer ships with citations back to the source chunks, the model is allowed to say "I don’t know" when retrieval comes up thin, and an eval harness tracks retrieval quality (did we fetch the right chunks?) and answer faithfulness (did the answer stick to them?). That’s how you catch a silently drifting index before users do.

RAG fails quietly: When RAG breaks, nothing errors — the answer just goes subtly wrong. Citations make it auditable; "I don’t know" makes it honest; evals make the failure visible instead of silent.

The payoff

You built a RAG pipeline

From a forgetful model to a grounded, citable system: an ingestion pipeline that chunks and embeds, a vector store for semantic search, two-stage retrieval with reranking, grounded generation, and evals to keep it honest.

Now stale the index and watch RAG’s signature failure — confident answers built on outdated chunks — and see why ingestion is a first-class, ongoing system, not a one-time script.

• RAG Gateway — orchestrates retrieve → augment → generate
• Ingestion — chunk + embed documents, offline and ongoing
• Embeddings — meaning as coordinates — same model for chunks + queries
• Vector Store — ANN search returns top-k by similarity in ms
• Top-k — wide for recall, then narrowed for precision
• Reranker — cross-encoder sharpens the shortlist to the best few
• Grounded prompt — answer only from context, cite, decline if thin
• Evals + citations — make a quiet failure mode visible and auditable