Retrieval-Augmented
Generation, explained.

Rungs 1 to 7 find documents. A user with a question then has to read them. And a language model asked the same question answers from its parameters, which is a fixed, undated, unattributable store — it cannot be updated without training, and it cannot say where anything came from.

The change is to wire the two together and train them as one system: retrieve passages with a dense retriever, condition a generator on the question and the retrieved text, and treat the retrieved document as a latent variable — so the generator's loss propagates back into the retriever, teaching it to fetch what actually helps generation rather than what looks similar to the question.

The property that follows is the one the whole industry took: the model's knowledge lives in an index you can edit. Swap a document and the answer changes with no gradient step; the answer can cite the passage it came from; and a claim can be checked against a source. That is a different kind of system from one that has memorised, and it is why this rung is the architecture behind essentially every deployed question-answering product. The paper also distinguishes marginalising over documents once per sequence from doing it per token — the latter lets a single answer draw on several passages, which matters exactly where one passage does not contain the whole answer.

By 2020, large language models could answer trivia surprisingly well — from memory alone. All their knowledge was crammed into their weights during training, a process the authors call parametric memory. Impressive, but it has three deep flaws.

It's lossy: facts get blurred into a statistical haze, so the model confidently makes things up — hallucinates. It's frozen: to teach it something new, or fix a wrong fact, you have to retrain. And it's opaque: it can't tell you where an answer came from, because there is no "where" — just a fog of weights.

The paper's fix is elegant: don't force the model to memorize everything. Pair it with a searchable external collection — the authors used all of Wikipedia — that it can look things up in on demand. They call this searchable index non-parametric memory.

Now answering a question becomes two steps instead of one: retrieve the most relevant documents, then generate an answer grounded in them. The model no longer has to be the encyclopedia; it just has to know how to read one.

Old-school search matched keywords — miss the exact word and you miss the document. RAG uses dense retrieval (DPR): a neural encoder turns the query into a vector and every document into a vector, and "relevant" simply means "close in vector space." Ask about "the capital of France" and it finds the passage about Paris even if the word "capital" never appears.

All of Wikipedia is pre-encoded once into a vector index. At query time, the retriever encodes the question and pulls the top-k nearest passages — a handful of the most relevant pages, in milliseconds.

The retrieved passages are handed to a generator — a sequence-to-sequence model (BART in the paper) — alongside the original question. The generator writes its answer conditioned on that evidence, effectively reading the passages and composing a response instead of dredging one up from memory.

Crucially, the query encoder and generator are trained together. The generator's success signal flows back and teaches the query side of the retriever what "relevant" really means for producing good answers. (The document encoder and its index stay frozen.)

The paper offers two flavors. RAG-Sequence commits to one set of retrieved documents and writes the entire answer from them — simple and coherent. RAG-Token is more fluid: it can lean on a different document for each token it generates, so a single answer can stitch together facts from several sources.

Ask "Where were these two authors born?" and RAG-Token can pull one author's birthplace from one passage and the other's from another, mid-sentence. More flexible. (RAG-Token is actually the cheaper one to decode; RAG-Sequence needs a beam search per document.)

The difference is exactly where the sum over documents sits. Both marginalize over the retrieved set z, but RAG-Sequence sums outside the product over answer tokens; RAG-Token sums inside:

That placement is the whole story. RAG-Sequence forces one document to account for every token, so an answer whose facts are split across two passages is penalized. RAG-Token lets each token pick its own document, so the split answer stays likely. The runnable version below puts numbers on the two-authors case — RAG-Token scores it ~2.8× higher.

Retrieval-Augmented Generation (Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal and colleagues, NeurIPS 2020) joins a pretrained retriever to a pretrained generator and fine-tunes them on each downstream task, without ever telling the model which document to retrieve. Two decisions define it: which parts are trained, and where the sum over documents sits.

1. Build the non-parametric memory once. Take the December 2018 Wikipedia dump, split it into disjoint 100-word chunks — 21 million documents — and encode each with DPR's document encoder, a BERT-base model, into a 768-dimensional vector held in a FAISS index with an HNSW approximation for fast maximum-inner-product search.
2. Encode the query. DPR's query encoder, also BERT-base, maps the input x to q(x). The retriever's distribution is p_η(z | x) ∝ exp(d(z) · q(x)), taken over the top k documents the index returns. Both encoders start from the DPR model trained on Natural Questions and TriviaQA.
3. Generate once per document. For each of the k retrieved documents z, the generator — BART-large, about 400 million parameters — reads the input and the document concatenated and produces token probabilities p_θ(y_i | x, z, y_1:i−1). That is k forward passes, run as a batch.
4. Marginalise. RAG-Sequence weights each document's whole-answer probability by p_η(z | x) and sums; RAG-Token does the same sum separately at every token, so each token can lean on a different document.
5. Train on input-output pairs only. Minimise the negative marginal log-likelihood of the target with Adam, using k = 5 or 10 documents. The gradient reaches the query encoder through p_η(z | x) and the generator directly. The document encoder and the index stay frozen: updating them would mean re-encoding 21 million passages as training goes, as REALM does during pretraining, and the authors found that unnecessary for strong results.
6. Decode RAG-Token with ordinary beam search. Its next-token distribution is already a single mixture, Σ_z p_η(z | x) · p_θ(y_i | x, z, y_1:i−1), so a standard beam decoder runs on it unchanged.
7. Decode RAG-Sequence per document. Its likelihood does not split into per-token terms, so beam search runs once per retrieved document, each hypothesis scored with that document alone, and the union of hypotheses is then rescored. Thorough Decoding runs extra forward passes so every hypothesis is scored under every document; Fast Decoding treats a hypothesis that never appeared in a document's beam as having near-zero probability there and skips those passes.
8. Swap the index at will. Because the memory is only an index of vectors made by a frozen encoder, it can be rebuilt from a newer dump and plugged in without retraining the query encoder or BART.

The parameter count shows the bargain: two 110-million-parameter BERT encoders and a 406-million-parameter BART, about 626 million in all, against 11 billion for the largest closed-book T5 it was compared with. The rest of the knowledge sits in 21 million vectors that cost nothing to train.

• d(z), q(x) — document and query vectors from DPR's two BERT encoders; d(z) is fixed, q(x) is trained.
• pη(z | x) — the retriever's prior over the k retrieved documents.
• pθ(y | x, z) — BART's probability of the whole target given the input and one document.
• p(z | x, y) — the posterior: the share of the target's likelihood that document z explains.

The gradient line is derived here from the paper's loss by differentiating through the softmax, and it says what training does to retrieval. The query vector moves toward documents whose posterior share exceeds their prior share — documents that made the right answer more likely than the retriever expected — and away from the rest. No document label is ever needed. At the extremes: with k = 1 the prior is always 1, the bracket is zero, the retriever gets no gradient, and RAG-Sequence and RAG-Token become the same model; and if every retrieved document explains the answer equally well, the bracket is again zero and retrieval is left alone.

RAG was fine-tuned separately for each task with 5 or 10 retrieved documents and evaluated on open-domain question answering by exact match, abstractive answers on MS MARCO, Jeopardy question generation and FEVER fact verification. Baselines were closed-book T5, the retrieve-and-read systems REALM and DPR, and BART fine-tuned without retrieval.

That is a new state of the art on three open-domain sets; on TriviaQA's standard open test split the best DPR pipeline stayed slightly ahead, 57.9 to 56.8. Generation also earned its keep: RAG produced a correct answer for 11.8% of Natural Questions cases where no retrieved document contained it, which an extractive reader cannot do. On FEVER, with no supervision about which evidence to retrieve, it reached 72.5% three-way accuracy against 76.8% for pipeline systems trained with intermediate retrieval supervision. On Jeopardy questions, human judges found RAG more factual than BART in 42.7% of comparisons and BART more factual in 7.1%.

The ablations are informative. Swapping the learned retriever for BM25 cost a lot on question answering but was slightly better on FEVER, whose claims are entity-heavy enough for word overlap to work; freezing the retriever was worse than learning it on every task. Retrieving more documents at test time kept helping RAG-Sequence on question answering, while RAG-Token peaked at 10. And the index swap: on questions about world leaders who changed between 2016 and 2018, the model answered 70% correctly with a 2016 index for 2016 leaders and 68% with a 2018 index for 2018 leaders, but only 12% and 4% with the indexes mismatched — the knowledge followed the index, with no retraining.

The reduction in fabrication is partial and not a guarantee, which the paper is careful about and the marketing was not. Grounding a generator in retrieved text makes it more likely to say what the text says; nothing forces it to. A model can still misread a passage, blend it with parametric knowledge, or produce a citation that does not support the sentence attached to it — and that last failure is worse than an uncited error, because it comes with a credential.

The system also inherits every weakness of its retriever. If the passage is not retrieved the answer cannot be grounded in it, and the retriever's out-of-domain fragility from rung 6 becomes the whole system's fragility. In practice most disappointing deployments of this architecture are retrieval failures wearing a generation costume.

How it held up: it is the default architecture for putting a language model on top of a corpus, and the reason the vector-database industry exists. What changed is that the joint training this paper introduced is mostly not done any more — the common deployment is an off-the-shelf retriever and an off-the-shelf model glued by a prompt, which is simpler and gives up the paper's most interesting mechanism. And the top-k framing has a structural limit that rung 9 names: some questions are not answered by any k passages.

Every "chat with your PDF," every enterprise knowledge assistant, every AI search box that cites its sources is a descendant of this paper. The specific models have changed — bigger generators, better embedding models, hybrid keyword-plus-vector retrieval, re-rankers, agentic multi-hop retrieval — but the core loop is unchanged: retrieve, then generate.

RAG reframed knowledge as something a model looks up rather than something it is. That separation — reasoning in the weights, facts in an index you control — is the single most important pattern in applied AI today.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

The longest ladder here, and the most linear: nine papers that carry one idea — text as geometry — from a 2013 word embedding to a graph-structured retrieval system.

1. 1Word2Vec2013
2. 2GloVe2014
3. 3Sentence-BERT2019
4. 4Product Quantization2011
5. 5HNSW2016
6. 6Dense Passage Retrieval2020
7. 7ColBERT2020
8. 8Retrieval-Augmented Generation2020
9. 9GraphRAG2024

All 110 papers, in 19 ladders →