Retrieval-augmented
generation, grounded.

A large language model answers from its parametric memory — everything baked into its weights during training. That has three hard limits: it's frozen at the training cutoff (it can't know recent events), it can't see your private data (your docs, your database), and when it doesn't know, it hallucinates a plausible answer with no sources to check.

Retrieval-augmented generation flips the order: before answering, retrieve the most relevant documents from an external source, put them in the prompt, and instruct the model to answer from that context and cite it. Knowledge now comes from retrieval — fresh, private, and attributable — while the model does what it's good at: reading and synthesizing. It's the difference between "recall from memory" and "open-book with the right pages already found."

RAG has an offline index-time phase and an online query-time phase. Index-time prepares the corpus once (and on updates); query-time runs per question.

Every step below is one of these boxes. Get any one wrong and the answer suffers: bad chunks or bad retrieval mean the model never sees the right passage, and no amount of clever prompting recovers information that isn't in the context.

You can't embed a whole document as one vector — models have input limits, and cramming a long document into a single vector blurs its many topics into mush. So documents are split into chunks (passages). This is quietly one of the highest-leverage decisions in RAG, because retrieval and the final answer both operate at chunk granularity.

Chunk size is a tradeoff. Too large and a chunk covers several topics, so its embedding is unfocused and retrieval is imprecise (and it wastes context). Too small and it loses the surrounding context needed to make sense. Add overlap between chunks so a sentence split across a boundary isn't orphaned, and prefer structure-aware splitting (by heading, paragraph, or semantic boundary) over blind fixed-size cuts, so each chunk is a coherent unit of meaning.

Each chunk is turned into an embedding — a vector that places its meaning in a space where similar meanings sit close together. Store these in a vector database that supports fast approximate nearest-neighbor search. At query time, the question is embedded with the same model, and the store returns the chunks whose vectors are closest — semantic search that matches on meaning, not keywords, so "reset my password" finds a doc titled "recovering account access."

Embeddings are the bridge from text to retrievable geometry. Their quality (and consistency) sets the ceiling on how good retrieval can be.

Given the query embedding, retrieve the top-k most similar chunks. But pure vector search has a blind spot: it matches meaning, so it can miss exact terms — a product code, a rare name, an error string that must match literally. So strong RAG uses hybrid retrieval: combine vector search (semantic) with keyword/BM25 search (exact terms) and fuse the results (e.g. reciprocal rank fusion).

The first stage optimizes for recall: make sure the truly relevant chunks are somewhere in the candidate set, even if not perfectly ordered — because the next stage can only reorder what retrieval found. Pull a generous candidate set (dozens to a couple hundred), wide enough to almost certainly contain the answer.

Retrieval finds relevant chunks but orders them roughly — because the embedding of the query and of each chunk are computed separately (a bi-encoder), so subtle query-document relevance is missed. Since the LLM only reads the top few chunks, ordering matters enormously. So add a second stage: a reranker.

A reranker is a cross-encoder: it looks at the query and a candidate chunk together and outputs a precise relevance score, far more accurate than comparing separate vectors — but too slow to run over the whole corpus. That's why it's a second stage: retrieve wide and cheap (recall), then rerank the top ~100 candidates and keep the best few (precision). This "retrieve wide → rerank narrow" pattern is often the single biggest quality win in a RAG system.

Now the payoff. Put the reranked chunks into the prompt and instruct the model to answer only from the provided context, attach an inline citation to every claim, and say "the sources don't cover this" rather than guess when the context is missing. Grounding must drive generation — answer from the evidence, citing as you go — not be bolted on afterward.

This is what turns an LLM into a trustworthy answer engine: every sentence is traceable to a source the user can check. Refusing when the evidence is absent is a feature, not a bug — it's the line between "I don't have a source for that" and a confident hallucination. Because the model can still misattribute, high-stakes systems add a verification step that checks each citation actually supports its claim.

Evaluate retrieval and generation independently, because a bad answer could come from either. Retrieval: metrics like recall and precision — did the relevant chunks get retrieved, and how highly ranked? If retrieval misses, nothing downstream can fix it. Generation: groundedness / faithfulness (is every claim supported by the retrieved context, or did the model hallucinate?) and answer relevance (did it actually address the question?). LLM-as-judge is commonly used to score these at scale.

Crucially, RAG quality drifts: the corpus changes, new documents arrive, embeddings age, and query patterns shift. So evaluation is continuous, not a one-time check — a quietly stale index degrades answers with no error message. Treat the eval harness as part of the system, not an afterthought.