Retrieval-Augmented
Generation, explained.

Rungs 1 to 7 find documents. A user with a question then has to read them. And a language model asked the same question answers from its parameters, which is a fixed, undated, unattributable store — it cannot be updated without training, and it cannot say where anything came from.

The change is to wire the two together and train them as one system: retrieve passages with a dense retriever, condition a generator on the question and the retrieved text, and treat the retrieved document as a latent variable — so the generator's loss propagates back into the retriever, teaching it to fetch what actually helps generation rather than what looks similar to the question.

The property that follows is the one the whole industry took: the model's knowledge lives in an index you can edit. Swap a document and the answer changes with no gradient step; the answer can cite the passage it came from; and a claim can be checked against a source. That is a different kind of system from one that has memorised, and it is why this rung is the architecture behind essentially every deployed question-answering product. The paper also distinguishes marginalising over documents once per sequence from doing it per token — the latter lets a single answer draw on several passages, which matters exactly where one passage does not contain the whole answer.

By 2020, large language models could answer trivia surprisingly well — from memory alone. All their knowledge was crammed into their weights during training, a process the authors call parametric memory. Impressive, but it has three deep flaws.

It's lossy: facts get blurred into a statistical haze, so the model confidently makes things up — hallucinates. It's frozen: to teach it something new, or fix a wrong fact, you have to retrain. And it's opaque: it can't tell you where an answer came from, because there is no "where" — just a fog of weights.

The paper's fix is elegant: don't force the model to memorize everything. Pair it with a searchable external collection — the authors used all of Wikipedia — that it can look things up in on demand. They call this searchable index non-parametric memory.

Now answering a question becomes two steps instead of one: retrieve the most relevant documents, then generate an answer grounded in them. The model no longer has to be the encyclopedia; it just has to know how to read one.

Old-school search matched keywords — miss the exact word and you miss the document. RAG uses dense retrieval (DPR): a neural encoder turns the query into a vector and every document into a vector, and "relevant" simply means "close in vector space." Ask about "the capital of France" and it finds the passage about Paris even if the word "capital" never appears.

All of Wikipedia is pre-encoded once into a vector index. At query time, the retriever encodes the question and pulls the top-k nearest passages — a handful of the most relevant pages, in milliseconds.

The retrieved passages are handed to a generator — a sequence-to-sequence model (BART in the paper) — alongside the original question. The generator writes its answer conditioned on that evidence, effectively reading the passages and composing a response instead of dredging one up from memory.

Crucially, retriever and generator are trained together, end to end. The generator's success signal flows back and teaches the retriever what "relevant" really means for producing good answers — the two halves learn to cooperate.

The paper offers two flavors. RAG-Sequence commits to one set of retrieved documents and writes the entire answer from them — simple and coherent. RAG-Token is more fluid: it can lean on a different document for each token it generates, so a single answer can stitch together facts from several sources.

Ask "Where were these two authors born?" and RAG-Token can pull one author's birthplace from one passage and the other's from another, mid-sentence. More flexible, slightly more expensive.

The difference is exactly where the sum over documents sits. Both marginalize over the retrieved set z, but RAG-Sequence sums outside the product over answer tokens; RAG-Token sums inside:

That placement is the whole story. RAG-Sequence forces one document to account for every token, so an answer whose facts are split across two passages is penalized. RAG-Token lets each token pick its own document, so the split answer stays likely. The runnable version below puts numbers on the two-authors case — RAG-Token scores it ~2.8× higher.

The reduction in fabrication is partial and not a guarantee, which the paper is careful about and the marketing was not. Grounding a generator in retrieved text makes it more likely to say what the text says; nothing forces it to. A model can still misread a passage, blend it with parametric knowledge, or produce a citation that does not support the sentence attached to it — and that last failure is worse than an uncited error, because it comes with a credential.

The system also inherits every weakness of its retriever. If the passage is not retrieved the answer cannot be grounded in it, and the retriever's out-of-domain fragility from rung 6 becomes the whole system's fragility. In practice most disappointing deployments of this architecture are retrieval failures wearing a generation costume.

How it held up: it is the default architecture for putting a language model on top of a corpus, and the reason the vector-database industry exists. What changed is that the joint training this paper introduced is mostly not done any more — the common deployment is an off-the-shelf retriever and an off-the-shelf model glued by a prompt, which is simpler and gives up the paper's most interesting mechanism. And the top-k framing has a structural limit that rung 9 names: some questions are not answered by any k passages.

RAG set new state-of-the-art on open-domain question answering — Natural Questions, TriviaQA, WebQuestions — beating both pure parametric models and earlier retrieve-and-read pipelines. Just as telling: its answers were more specific, more factual, and less prone to hallucination, and because every answer traced back to real passages, they could be cited.

That last row is the quiet superpower: swap in a fresh Wikipedia dump — or fix one wrong document — and the model's knowledge updates immediately, no retraining required.

Every "chat with your PDF," every enterprise knowledge assistant, every AI search box that cites its sources is a descendant of this paper. The specific models have changed — bigger generators, better embedding models, hybrid keyword-plus-vector retrieval, re-rankers, agentic multi-hop retrieval — but the core loop is unchanged: retrieve, then generate.

RAG reframed knowledge as something a model looks up rather than something it is. That separation — reasoning in the weights, facts in an index you control — is the single most important pattern in applied AI today.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

The longest ladder here, and the most linear: nine papers that carry one idea — text as geometry — from a 2013 word embedding to a graph-structured retrieval system.

1. 1Word2Vec2013
2. 2GloVe2014
3. 3Sentence-BERT2019
4. 4Product Quantization2011
5. 5HNSW2016
6. 6Dense Passage Retrieval2020
7. 7ColBERT2020
8. 8Retrieval-Augmented Generation2020
9. 9GraphRAG2024

All 110 papers, in 19 ladders →