Retrieval-Augmented
Generation, explained.

By 2020, large language models could answer trivia surprisingly well — from memory alone. All their knowledge was crammed into their weights during training, a process the authors call parametric memory. Impressive, but it has three deep flaws.

It's lossy: facts get blurred into a statistical haze, so the model confidently makes things up — hallucinates. It's frozen: to teach it something new, or fix a wrong fact, you have to retrain. And it's opaque: it can't tell you where an answer came from, because there is no "where" — just a fog of weights.

The paper's fix is elegant: don't force the model to memorize everything. Pair it with a searchable external collection — the authors used all of Wikipedia — that it can look things up in on demand. They call this searchable index non-parametric memory.

Now answering a question becomes two steps instead of one: retrieve the most relevant documents, then generate an answer grounded in them. The model no longer has to be the encyclopedia; it just has to know how to read one.

Old-school search matched keywords — miss the exact word and you miss the document. RAG uses dense retrieval (DPR): a neural encoder turns the query into a vector and every document into a vector, and "relevant" simply means "close in vector space." Ask about "the capital of France" and it finds the passage about Paris even if the word "capital" never appears.

All of Wikipedia is pre-encoded once into a vector index. At query time, the retriever encodes the question and pulls the top-k nearest passages — a handful of the most relevant pages, in milliseconds.

The retrieved passages are handed to a generator — a sequence-to-sequence model (BART in the paper) — alongside the original question. The generator writes its answer conditioned on that evidence, effectively reading the passages and composing a response instead of dredging one up from memory.

Crucially, retriever and generator are trained together, end to end. The generator's success signal flows back and teaches the retriever what "relevant" really means for producing good answers — the two halves learn to cooperate.

The paper offers two flavors. RAG-Sequence commits to one set of retrieved documents and writes the entire answer from them — simple and coherent. RAG-Token is more fluid: it can lean on a different document for each token it generates, so a single answer can stitch together facts from several sources.

Ask "Where were these two authors born?" and RAG-Token can pull one author's birthplace from one passage and the other's from another, mid-sentence. More flexible, slightly more expensive.

RAG set new state-of-the-art on open-domain question answering — Natural Questions, TriviaQA, WebQuestions — beating both pure parametric models and earlier retrieve-and-read pipelines. Just as telling: its answers were more specific, more factual, and less prone to hallucination, and because every answer traced back to real passages, they could be cited.

That last row is the quiet superpower: swap in a fresh Wikipedia dump — or fix one wrong document — and the model's knowledge updates immediately, no retraining required.

Every "chat with your PDF," every enterprise knowledge assistant, every AI search box that cites its sources is a descendant of this paper. The specific models have changed — bigger generators, better embedding models, hybrid keyword-plus-vector retrieval, re-rankers, agentic multi-hop retrieval — but the core loop is unchanged: retrieve, then generate.

RAG reframed knowledge as something a model looks up rather than something it is. That separation — reasoning in the weights, facts in an index you control — is the single most important pattern in applied AI today.