Why retrieval at all?

Retrieval-Augmented Generation: before generating, fetch the most relevant passages from your knowledge and put them in the prompt. The model reasons over real, current text instead of fuzzy memory — and can cite its sources.

Retrieve, then generate

A RAG Gateway orchestrates two phases for every question: retrieve the most relevant passages, then generate an answer grounded in them. Knowledge lives outside the model, so it’s always current and citable.

Chunk and embed your documents

An offline Ingestion pipeline loads each document, splits it into chunks, runs every chunk through an Embedding Model to get a vector, and upserts the vectors into a Vector Store (plus the raw text into a Document Store). This is the prep work that makes retrieval possible.

Embed the query, search by similarity

The Retriever embeds the query with the same model, then asks the Vector Store for the top-k nearest chunks via an ANN index. The Document Store returns their text. Semantic match means "cancel my plan" finds "terminating your subscription."

Top-k and the recall/precision dial

Retrieve a modest top-k candidate set from the Vector + Document stores — wide enough that the answer is almost certainly in there (recall), but not so wide it drowns the prompt. The next step tightens it to the best few (precision).

A Reranker (a cross-encoder) re-scores the top-k candidates by reading the query and each chunk together, then keeps only the best 3–5. It’s too slow to run over the whole store — but perfect over a 20-candidate shortlist.

Build the prompt and generate

The Gateway assembles a grounded prompt — system instructions + the reranked chunks + the question — and tells the LLM to answer only from the passages, cite them, and admit when they don’t cover the question. The chunks carry their source metadata, so citations point at real documents.

Citations, "I don’t know", and evals

The answer ships with citations back to the source chunks, the model is allowed to say "I don’t know" when retrieval comes up thin, and an eval harness tracks retrieval quality (did we fetch the right chunks?) and answer faithfulness (did the answer stick to them?). That’s how you catch a silently drifting index before users do.

Design a RAG Pipeline — A Guided System Design

System Design · step by stepDesign a RAG Pipeline

Step 1 / 9

Finished this one? 0 / 5 AI System Designs done

More AI System Designs

Design a Conversational AIBuild a production conversational AI system (think ChatGPT) step by step. See how the request path splits an inference gateway from the model servers, how the context window is assembled and token-budgeted, how conversation memory is stored and recalled, how tokens stream back over a persistent connection, and how guardrails gate every prompt and response — through an interactive diagram that grows with each concept.Read →
Design an AI Agent SystemBuild an autonomous AI agent step by step. See how the plan-act-observe loop turns a goal into action, how the model emits typed tool calls, how a sandboxed executor runs them safely, how working and long-term memory fit together, and how budgets and approval gates keep a multi-step agent from running away — through an interactive diagram that grows with each concept.Read →
Design an LLM Inference ServerBuild an LLM inference serving system step by step. See how a request queue absorbs spiky traffic, how the prefill/decode split and continuous batching keep GPUs full, how the KV cache and paged attention make each token cheap, how tensor sharding fits a giant model, and how autoscaling rides demand — all balancing latency against throughput, through an interactive diagram that grows with each concept.Read →
Design a Recommendation SystemBuild a large-scale recommendation system step by step. See how a two-stage retrieve-and-rank funnel picks the best few from millions, how two-tower embeddings and ANN generate candidates fast, how a heavy ranking model scores engagement, how a feature store stays consistent between training and serving, and how the feedback loop keeps recommendations fresh — through an interactive diagram that grows with each concept.Read →

Explore more from Vibe Engines

Handbooks →

Roadmaps →

Prompt Engineering Roadmap

System Designs →