RAG & Retrieval
Retrieval-augmented generation end to end — chunking, embeddings, similarity search and ranking — across a system design, a hands-on tool, and runnable challenges.
Handbooks 9
The RAG Handbook
Retrieval-augmented generation end to end — why LLMs need it, chunking, embeddings and vector search, hybrid retrieval, two-stage reranking, grounded generation with citations, and how to evaluate a RAG system so a quietly-drifting index never fools you.
The Vector Databases Handbook
How vector search actually works — why brute-force kNN doesn't scale, distance metrics, approximate nearest neighbor, the HNSW, IVF and product-quantization indexes and their recall/latency/memory tradeoffs, metadata filtering, updates, scaling, and when you need a dedicated vector DB vs pgvector.
RAG vs Fine-Tuning
The most common wrong turn in AI engineering — reaching for fine-tuning when you needed retrieval, or vice versa. The one distinction that decides it (knowledge vs behavior), a head-to-head table, when to use each, and why the strongest systems combine them: fine-tune for behavior, RAG for knowledge.
The Embeddings Handbook
Meaning as geometry: how text and images become vectors so that similar meaning lands nearby, why cosine similarity measures direction not length, how L2-normalization makes the dot product equal cosine (so vector databases just take dot products), the curse of dimensionality, and how it all powers semantic search and RAG — with worked math and runnable code.
The Data Engineering for AI Handbook
The pipeline that turns raw documents into usable data for RAG and training — ingest, chunk, deduplicate, quality-filter. How overlapping chunking works, why deduplication restores diversity, and why the usable dataset is a compounding fraction of the raw pile (raw × dedup-rate × quality-rate) — because quality beats quantity for both retrieval and training. With worked yield math and a runnable pipeline.
RAG vs Long Context
A 1M-token window makes it tempting to skip retrieval entirely — but bigger context doesn’t mean the model reads it evenly, and it isn’t cheap. The "lost in the middle" attention problem, cost-at-scale math, and the retrieve-then-go-long pattern that combines both.
Embeddings vs Keyword Search
Embeddings match meaning even with zero shared words; keyword search (BM25) matches exact terms with a precision embeddings can’t guarantee. Why the two fail in genuinely complementary ways, and why hybrid search with fusion is the modern RAG default.
AI App Patterns for FDEs
The AI toolkit a modern FDE deploys: RAG over the customer’s documents, agents that drive their workflows, and evals as the acceptance test — plus how to choose between prompting, RAG, agents, and fine-tuning, and how to ship AI honestly with a good-enough bar and a human in the loop.
Cross-Encoder vs Bi-Encoder
Cross-encoder vs bi-encoder for semantic search, decided by when the query meets the document: a bi-encoder embeds query and document separately so document vectors precompute and scale to millions; a cross-encoder feeds them together for a far more accurate relevance score, but runs per pair at query time. Why production retrieval uses a bi-encoder to retrieve and a cross-encoder to rerank.
System Designs 1
AI System Designs 20
Design a Conversational AI
Build a production conversational AI system (think ChatGPT). See how the request path splits an inference gateway from the model servers, how the context window is assembled and token-budgeted, how conversation memory is stored and recalled, how tokens stream back over a persistent connection, and how guardrails gate every prompt and response.
Design a RAG Pipeline
Build a retrieval-augmented generation pipeline. See how documents are chunked and embedded, how a vector store answers semantic search, how two-stage retrieval with reranking finds the best passages, how the prompt is grounded to stop hallucination, and how evals keep a quietly-drifting index honest.
Design a Vector Database
Build a vector database. See why "k nearest of a billion vectors" needs its own index, how a distance metric ranks similarity, how IVF cells and an HNSW graph make search sub-linear, how product quantization fits billions in RAM, how metadata filtering and sharding hold up — and how ANN fails silently when you starve the search.
Design Semantic Search
Build a semantic search engine. See why keyword search misses meaning, how a single shared embedding model puts documents and queries in one space, how the chunk→embed→index ingest path is built, how hybrid BM25 + vector fusion catches exact terms, how a cross-encoder reranks the shortlist — and how a model-version upgrade silently randomizes results.
Design an AI Coding Assistant
Build an AI coding assistant (like Copilot or Cursor). See how inline completion / tab prediction meets a sub-second latency budget, how context is assembled with fill-in-the-middle, how repo-aware retrieval grounds completions in the codebase, how a code-specialized model streams suggestions, how debounce/cancel/cache win the milliseconds, how acceptance-rate telemetry measures quality, how an agentic chat mode handles multi-file edits, and why starving the context yields confident wrong code — through an interactive diagram.
Design an AI Answer Engine
Build an AI answer engine like Perplexity. See why a bare LLM is stale and hallucinates with no sources, how retrieve-then-generate (RAG over the live web) grounds answers, how a query planner rewrites and decomposes questions, how search + fetch + rerank build a tight evidence set, how grounded generation cites every claim, how citation verification catches unsupported statements, and how to defend against prompt injection from web content.
Design an Embeddings Service
Build an embeddings service that powers semantic search, RAG and recommendations. See why embedding needs a shared service, how online (low-latency query) and batch (millions of docs) modes differ, what the model outputs, how a batched GPU pipeline indexes a corpus, the critical rule that query and corpus must share the same model version (so a model change re-embeds everything), and how caching by (text, version) saves cost.
Design a Reranking Service
Build a reranking service — the precision tier of two-stage retrieval for search and RAG. See why vector (bi-encoder) search has great recall but rough ordering, how retrieve-wide-then-rerank-narrow works, why a cross-encoder scores query and candidate jointly for precision, how the top-K latency/cost tradeoff is tuned, how to serve it batched and cached, and why recall gates precision so both stages are essential.
Design a GraphRAG System
Build a GraphRAG system — knowledge-graph-augmented retrieval. See why vanilla vector RAG fails on multi-hop and global questions, how a knowledge graph of entities and relationships lets retrieval traverse connections, how LLM extraction builds the graph, how community detection and hierarchical summaries enable global questions, how local vs global query modes work, and why hybrid graph+vector retrieval is the strongest form.
Design a Multimodal RAG System
Build a multimodal RAG system that answers questions grounded in images, charts, tables and PDFs. See why text-only RAG loses visual information, how documents are parsed into multimodal chunks, unified multimodal embeddings vs convert-to-text, cross-modal retrieval, why generation needs a vision-language model that actually sees the retrieved images, how assets are stored and served, and the modality-gap and cost tradeoffs.
Design an Agent Memory System
Build a long-term memory system for an LLM agent or chatbot. See why stuffing the whole history into context fails, how short-term (context window) and long-term (persistent) memory differ, how the write path extracts salient facts, how context is assembled under a token budget, how memories live in vector + structured stores, how recall is RAG over memory, and why consolidation and forgetting keep memory coherent.
Design a Deep Research Agent
Build a deep research agent — the system behind every "deep research" product. See how a lead agent decomposes the question into a coverage plan, why parallel sub-researchers get isolated contexts, how the search-read-note loop compounds understanding, why evidence lives as claims-paired-with-sources, how gap-driven iteration ends under a hard budget, and how citation verification keeps the final report honest — through an interactive diagram where you can poison a source.
Design an AI Meeting Notetaker
Build an AI meeting notetaker. See how a capture bot joins calls with per-speaker audio, why consent is architecture rather than a checkbox, how streaming ASR and diarization turn chaos into an attributed transcript, why extracted decisions and action items must cite timestamped segments, how meeting memory answers questions across months of calls, and how artifacts land in the tools where work happens — through an interactive diagram where you can unleash crosstalk.
Design an Adaptive AI Tutor
Build an adaptive tutoring system like Khanmigo or Duolingo's AI tutor. See why a fixed question order fails every learner, how a per-skill mastery model picks the next problem, why grading routes structured answers to a deterministic checker and reserves the LLM for open-ended judgment, how the mastery feedback loop closes, why explanations are grounded in a curriculum knowledge base instead of freely generated, how a Socratic hint ladder avoids just giving away the answer, and how a long-term learner profile drives spaced repetition.
Design an AI Code Review Bot
Build an automated PR review bot like CodeRabbit or Graphite. See why cheap deterministic linters run before any LLM call, how review is scoped to the diff plus just enough context, how a repo embedding index (RAG over the codebase) supplies cross-file context the diff alone can't show, why every finding is confidence-gated before posting, how a comment ledger prevents re-pushes from spamming old feedback, how secrets are redacted before they ever reach the LLM, and how developer reactions tune down false positives over time.
Design a Data Labeling Platform
Build a human annotation platform like Scale AI or Labelbox — sourcing labels from people, not generating them programmatically. See why a single annotator's answer can't be trusted as ground truth, multi-annotator consensus, secretly-mixed gold-standard items that measure labeler accuracy in real time, skill-based task routing, active learning to prioritize which unlabeled examples matter most, and aligning labeler pay with measured quality.
Design a Multi-Tenant AI SaaS Platform
Build an AI product serving many separate customer organizations from shared infrastructure. See why unscoped shared retrieval risks cross-tenant data leaks, enforcing isolation at the data layer, per-tenant customization via configuration, noisy-neighbor prevention with per-tenant quotas, per-tenant cost attribution, data-residency routing, automated onboarding, and defense-in-depth isolation that survives a bug in any single layer.
Design a Sales CRM Agent
Build an AI agent embedded in a CRM that enriches leads, scores priority, drafts outreach, and suggests next actions — writing into a shared system of record and staying a suggester, not an autonomous actor, for relationship-sensitive interactions. See multi-source enrichment with provenance tracking, lead scoring, write-back validation that fails closed, next-best-action suggestion, personalization at scale, activity logging, and stale-data safeguards.
Design a Personal Knowledge Assistant
Build an AI assistant grounded in a user's own private documents and notes, where privacy is the central architectural constraint. See why dumping years of personal content into one context window fails, retrieval over messy heterogeneous formats, continuous incremental ingestion, disambiguating with personal context, source attribution, encryption and minimizing third-party exposure, genuine deletion, and recency-aware retrieval for facts that change over time.
Design Secure Document Ingestion + RAG
Build a retrieval-augmented assistant over a customer’s sensitive internal documents, where security — not relevance — is the hard constraint. See why naive RAG leaks documents across users, classifying and tagging access on ingest, permission-aware retrieval that intersects relevance with what the user may see, redacting PII and secrets before the prompt and logs, defending against prompt injection carried inside untrusted documents, and citing sources plus auditing every access so the pipeline can pass a security review.
Paper Breakdowns 7
Retrieval-Augmented Generation
The paper behind every "chat with your docs" product. Pair a generator with a neural retriever so the model looks knowledge up instead of memorizing it — dense retrieval, top-k documents, RAG-Sequence vs RAG-Token, and why retrieval cuts hallucination.
ColBERT
Keep one vector per token, not one per document, and score by late interaction: each query token takes its best match anywhere in the document (MaxSim), summed. It recovers the term-level precision single-vector retrieval blurs away, at index-friendly speed. The math, the case where it clearly wins, and how it sits between dense and cross-encoder retrieval — with runnable code.
Sentence-BERT
The model that turned BERT into fast, comparable sentence embeddings — the ancestor of every embedding model behind vector search and RAG. Why plain BERT can’t be compared without a pass per pair, the siamese fix with mean pooling, and the O(n²)→O(n) arithmetic that took a task from 65 hours to 5 seconds. Worked math plus runnable code.
Product Quantization
The compression that makes billion-scale vector search fit in memory. Split a vector into subvectors, quantize each with a 256-entry codebook, and store a 512-byte vector in 8 — while m small codebooks span k^m codes. The compression math, asymmetric distance lookups, and IVFPQ — worked math plus runnable code.
GraphRAG
RAG for the global questions vector retrieval can't answer. GraphRAG builds a knowledge graph from the corpus, detects communities of related entities, summarizes each, and map-reduces those summaries — so "what are the main themes across everything?" gets a comprehensive answer. Why top-k retrieval under-covers global queries, and the graph pipeline — worked math plus runnable code.
Dense Passage Retrieval
The dual-encoder method that made dense retrieval beat BM25 for open-domain QA. DPR encodes questions and passages into a shared vector space and matches by dot product; its efficiency trick is in-batch negatives — a batch of B pairs yields a B×B similarity matrix whose diagonal is the positive and whose off-diagonal gives B×(B−1) negatives for free. Why matching on meaning beats keywords, and how the free-negatives trick trains it — worked math plus runnable code.
HNSW
The graph index behind fast approximate nearest-neighbor search — the default in nearly every vector database. HNSW wires points into a navigable small-world graph and searches it greedily (hop to the neighbor closest to the query), organized into a skip-list-style hierarchy of layers whose height grows like ln(N) — so search is O(log N) instead of O(N). Greedy navigation, the exponential level assignment, and why squaring the dataset only doubles the layers — worked math plus runnable code.
Coding Challenges 6
Cosine Similarity
The measure behind every embedding search and RAG system: how aligned are two vectors, ignoring their length? Dot product over the product of magnitudes — 1 identical, 0 orthogonal, -1 opposite. Solve it in Python or TypeScript.
Top-K Retrieval
The core of the "R" in RAG: given a query embedding and a set of document embeddings, return the indices of the k most similar docs by cosine similarity, with a stable tie-break. Solve it in Python or TypeScript.
TF-IDF
The scoring that ran search for decades — and still seeds hybrid retrieval today. Reward a word frequent in one document but rare across the corpus, shrug off words that appear everywhere: tf·log(N/df). Solve it in Python or TypeScript.
BM25 Scoring
The keyword-ranking function every fancy retriever still has to beat. Implement BM25 — term frequency that saturates, rare terms weighted up, long documents normalized down — the lexical half of every hybrid search pipeline. Solve it in Python or TypeScript.
MMR Reranking
Top-k by similarity returns five copies of the same paragraph. Implement Maximal Marginal Relevance: greedily pick results that are relevant to the query AND different from what you already picked — λ·relevance − (1−λ)·redundancy, with cosine similarity. Solve it in Python or TypeScript.
Semantic Chunker (Token Budget)
Before you embed a document for retrieval, split it into chunks that fit a token budget — without slicing a sentence. The greedy packer fills each chunk with whole sentences until the next would overflow. Solve it in Python or TypeScript, with hidden tests.
Labs 1
Interactive Tools 2
RAG Chunking Playground
Drop in any text and compare chunking strategies — fixed-size, recursive, by-sentence, by-paragraph — with overlap highlighted and an estimated token count per chunk. Stop guessing your chunk size; see exactly how your RAG pipeline will split a document.
Embedding Cost Calculator
Set your corpus (documents × average tokens), chunking (size + overlap), embedding model and vector dtype — and get the numbers that decide RAG budgets: one-time indexing cost, vector count, storage footprint from FP32 down to binary, and the monthly bill for query embeddings.
About RAG & retrieval
Retrieval-augmented generation (RAG) is how you make a language model answer from your data instead of only its training. The pattern is simple to say and subtle to build: chunk your documents, embed each chunk into a vector, store the vectors, and at query time retrieve the closest ones and put them in the prompt so the model answers from real source material rather than hallucinating.
The quality lives in the details this topic walks through — how you chunk (too big blurs the meaning, too small loses context), which similarity metric ranks matches, how reranking sharpens the top results, and how evals catch a quietly-drifting index. Get retrieval right and the generator has little room to go wrong; get it wrong and no prompt engineering saves you.