What is Retrieval-Augmented Generation (RAG)?

RAG is a technique, introduced in a 2020 paper, that pairs a language model with a retriever over an external document collection. Instead of answering only from what it memorized during training, the model first fetches the most relevant documents for a query and then generates its answer conditioned on them — so knowledge lives in a searchable index it can look things up in, not just in frozen weights.

What is the difference between RAG and fine-tuning?

Fine-tuning bakes new knowledge into the model's weights through more training — expensive, slow to update, and hard to attribute. RAG keeps knowledge in an external index you can edit instantly: add, remove, or correct a document and the model's answers change immediately, with no retraining. Fine-tuning changes behavior and style; RAG changes what the model knows.

What is dense passage retrieval?

Dense retrieval encodes both the query and every document into vectors with a neural encoder, then finds the closest document vectors to the query vector — matching on meaning rather than exact keywords. The RAG paper used DPR (Dense Passage Retrieval) over a vector index of Wikipedia to fetch the top-k passages for each query.

What is the difference between RAG-Sequence and RAG-Token?

RAG-Sequence picks a set of retrieved documents and generates the whole answer conditioned on the same documents. RAG-Token can attend to different retrieved documents for different tokens of the answer, letting a single response weave together facts from several sources — more flexible, at a little more cost.

Paper Breakdowns / RAG

Paper 03~8 min readNeurIPS 2020

Paper Breakdown

Retrieval-Augmented
Generation, explained.

Q: Why does RAG reduce hallucination?

A pure language model answers from a blurry compression of its training data and will confidently invent facts it never really learned. RAG grounds each answer in specific retrieved passages, giving the model real evidence to quote and letting the system cite sources — which sharply reduces fabricated facts and makes answers checkable.

A language model is a brilliant student who took a closed-book exam on the entire internet — and now answers everything from memory, blurring details and inventing the rest. This 2020 paper handed that student a library card: retrieve the right pages first, then answer. It's the idea behind every "chat with your docs" product you've used.

Written breakdown

This one is a written deep-dive.

Paper, mechanism, worked math and a runnable version — all below ↓

The closed-book problem Give it a library card The retriever The generator RAG-Sequence vs RAG-Token The results Why it still matters

The closed-book problem

By 2020, large language models could answer trivia surprisingly well — from memory alone. All their knowledge was crammed into their weights during training, a process the authors call parametric memory. Impressive, but it has three deep flaws.

It's lossy: facts get blurred into a statistical haze, so the model confidently makes things up — hallucinates. It's frozen: to teach it something new, or fix a wrong fact, you have to retrain. And it's opaque: it can't tell you where an answer came from, because there is no "where" — just a fog of weights.

The big idea: give it a library card

The paper's fix is elegant: don't force the model to memorize everything. Pair it with a searchable external collection — the authors used all of Wikipedia — that it can look things up in on demand. They call this searchable index non-parametric memory.

Now answering a question becomes two steps instead of one: retrieve the most relevant documents, then generate an answer grounded in them. The model no longer has to be the encyclopedia; it just has to know how to read one.

The RAG loop

Query→ Retriever→ Top-k docs→ Generator→ Grounded answer

The retriever: matching on meaning

Old-school search matched keywords — miss the exact word and you miss the document. RAG uses dense retrieval (DPR): a neural encoder turns the query into a vector and every document into a vector, and "relevant" simply means "close in vector space." Ask about "the capital of France" and it finds the passage about Paris even if the word "capital" never appears.

All of Wikipedia is pre-encoded once into a vector index. At query time, the retriever encodes the question and pulls the top-k nearest passages — a handful of the most relevant pages, in milliseconds.

Worth knowing

This vector-search step is exactly what a vector database does today. The RAG paper is why "embed your documents, then do nearest-neighbor search" became the backbone of modern AI applications.

The generator: answer from evidence

The retrieved passages are handed to a generator — a sequence-to-sequence model (BART in the paper) — alongside the original question. The generator writes its answer conditioned on that evidence, effectively reading the passages and composing a response instead of dredging one up from memory.

Crucially, retriever and generator are trained together, end to end. The generator's success signal flows back and teaches the retriever what "relevant" really means for producing good answers — the two halves learn to cooperate.

RAG-Sequence vs RAG-Token

The paper offers two flavors. RAG-Sequence commits to one set of retrieved documents and writes the entire answer from them — simple and coherent. RAG-Token is more fluid: it can lean on a different document for each token it generates, so a single answer can stitch together facts from several sources.

Ask "Where were these two authors born?" and RAG-Token can pull one author's birthplace from one passage and the other's from another, mid-sentence. More flexible, slightly more expensive.

The difference is exactly where the sum over documents sits. Both marginalize over the retrieved set z, but RAG-Sequence sums outside the product over answer tokens; RAG-Token sums inside:

That placement is the whole story. RAG-Sequence forces one document to account for every token, so an answer whose facts are split across two passages is penalized. RAG-Token lets each token pick its own document, so the split answer stays likely. The runnable version below puts numbers on the two-authors case — RAG-Token scores it ~2.8× higher.

RUN IT YOURSELF

Where RAG-Sequence and RAG-Token split

Retrieval is an inner-product top-k with a softmax prior; the two marginalizations are three lines each. The demo is the paper's own "where were these two authors born?" case: doc A knows author one's birthplace, doc B knows author two's, neither knows both. RAG-Sequence — which makes a single document explain the whole answer — scores it 0.09; RAG-Token, free to switch documents per token, scores it 0.25, about 2.8× higher. That gap is the reason RAG-Token exists. Change the per-document token probabilities and watch which scheme wins.

CPython · WebAssembly

import math

# Both schemes marginalize over retrieved docs z; they differ only in WHERE the
# sum sits. RAG-Sequence: SUM_z outside the token product. RAG-Token: SUM_z inside.

def dot(a, b): return sum(x * y for x, y in zip(a, b))

def softmax(scores):
    m = max(scores)
    exps = [math.exp(s - m) for s in scores]
    return [e / sum(exps) for e in exps]

def retrieve(query, docs, k):                     # docs: [(name, <a href="/glossary/embedding" class="ve-gloss">embedding</a>), ...]
    ranked = sorted(docs, key=lambda d: dot(query, d[1]), reverse=True)[:k]
    names = [n for n, _ in ranked]
    priors = softmax([dot(query, e) for _, e in ranked])
    return names, priors

def rag_sequence(priors, token_probs):            # SUM_z p(z) * PROD_i p(y_i|z)
    total = 0.0
    for pz, probs in zip(priors, token_probs):
        seq = 1.0
        for p in probs:
            seq *= p
        total += pz * seq
    return total

def rag_token(priors, token_probs):               # PROD_i SUM_z p(z) * p(y_i|z)
    n = len(token_probs[0])
    prod = 1.0
    for i in range(n):
        prod *= sum(pz * token_probs[z][i] for z, pz in enumerate(priors))
    return prod

names, priors_r = retrieve([1, 1], [("A", [3.0, 0.0]), ("B", [0.0, 2.9]), ("C", [0.5, 0.5])], 2)
print(f"top-2 retrieved : {names}   priors {[round(p, 3) for p in priors_r]}\n")

priors = [0.5, 0.5]
token_probs = [[0.9, 0.1],      # doc A: nails author-1 birthplace, weak on author-2
               [0.1, 0.9]]      # doc B: opposite
seq, tok = rag_sequence(priors, token_probs), rag_token(priors, token_probs)
print(f"RAG-Sequence p(answer) : {seq:.3f}   (one doc must explain both facts)")
print(f"RAG-Token    p(answer) : {tok:.3f}   ({tok / seq:.1f}x higher -- switches docs per token)")

The real results

RAG set new state-of-the-art on open-domain question answering — Natural Questions, TriviaQA, WebQuestions — beating both pure parametric models and earlier retrieve-and-read pipelines. Just as telling: its answers were more specific, more factual, and less prone to hallucination, and because every answer traced back to real passages, they could be cited.

Approach	Knowledge lives in…	Update knowledge
Closed-book LLM	Frozen weights (parametric)	Retrain the model
Fine-tuning	Frozen weights (parametric)	Retrain / re-tune
RAG	Editable index (non-parametric)	Edit a document — instant

That last row is the quiet superpower: swap in a fresh Wikipedia dump — or fix one wrong document — and the model's knowledge updates immediately, no retraining required.

Why it still matters

Every "chat with your PDF," every enterprise knowledge assistant, every AI search box that cites its sources is a descendant of this paper. The specific models have changed — bigger generators, better embedding models, hybrid keyword-plus-vector retrieval, re-rankers, agentic multi-hop retrieval — but the core loop is unchanged: retrieve, then generate.

RAG reframed knowledge as something a model looks up rather than something it is. That separation — reasoning in the weights, facts in an index you control — is the single most important pattern in applied AI today.

Worth knowing

Building a RAG system today means choosing an embedding model, a vector database, a chunking strategy, and a re-ranker — then evaluating retrieval quality. Every one of those decisions traces back to the two-step loop this paper introduced.

Frequently asked

Quick answers

What is Retrieval-Augmented Generation?

Pairing a language model with a retriever over an external document index, so it fetches relevant documents for a query and answers grounded in them — knowledge lives in a searchable index, not only in weights.

Why does RAG reduce hallucination?

It grounds answers in specific retrieved passages the model can quote and the system can cite, instead of dredging facts from a lossy memory — so answers are checkable and far less likely to be invented.

RAG or fine-tuning?

Fine-tuning bakes knowledge into weights (expensive, slow to update). RAG keeps knowledge in an editable index you can change instantly. Fine-tuning changes behavior/style; RAG changes what the model knows.

What is dense retrieval?

Encoding query and documents into vectors and matching on closeness in vector space — meaning, not keywords. The paper used DPR over a vector index of Wikipedia to fetch the top-k passages.

RAG-Sequence vs RAG-Token?

RAG-Sequence writes the whole answer from one set of documents; RAG-Token can attend to a different document per token, weaving facts from several sources into one response.

Finished this one? 0 / 111 Paper Breakdowns done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

RAG & Retrieval

Retrieval-Augmented
Generation, explained.

The closed-book problem

The big idea: give it a library card

The retriever: matching on meaning

The generator: answer from evidence

RAG-Sequence vs RAG-Token

Where RAG-Sequence and RAG-Token split

The real results

Why it still matters

Quick answers

What is Retrieval-Augmented Generation?

Why does RAG reduce hallucination?

RAG or fine-tuning?

What is dense retrieval?

RAG-Sequence vs RAG-Token?

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Retrieval-AugmentedGeneration, explained.

The closed-book problem

The big idea: give it a library card

The retriever: matching on meaning

The generator: answer from evidence

RAG-Sequence vs RAG-Token

Where RAG-Sequence and RAG-Token split

The real results

Why it still matters

Quick answers

What is Retrieval-Augmented Generation?

Why does RAG reduce hallucination?

RAG or fine-tuning?

What is dense retrieval?

RAG-Sequence vs RAG-Token?

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Retrieval-Augmented
Generation, explained.