DPR,
explained.

Rungs 1 to 3 produce general-purpose vectors, and rungs 4 and 5 index them efficiently. None of them is trained on the task. For question answering that is a real mismatch, because a question and the passage that answers it are often not similar texts at all — they share little vocabulary and differ in form. Which is why keyword scoring had remained a stubbornly strong baseline: general similarity is not the relation you want.

The change is to train two encoders jointly on question-passage pairs — one for questions, one for passages, into a shared space where the inner product is high exactly when the passage answers the question. A learned relation, not a generic one, and a clean demonstration (after earlier work such as ORQA) that dense retrieval beats strong keyword matching on open-domain question answering.

The training trick is the rung's most reusable content, because the obvious obstacle is negatives: a contrastive objective needs wrong answers, and collecting them is expensive. The observation is that a batch of pairs already contains them — for any question, the passages belonging to the other questions in the batch are almost certainly not its answer. A batch of B pairs therefore yields B×(B−1) negatives for free, and the whole comparison is one matrix multiplication of the batch against itself. The vision ladder's contrastive rung is the same accounting applied to crops of an image, and it is why both papers care so much about batch size.

Open-domain question answering starts with a retrieval step: pull a handful of relevant passages from millions, then read them for the answer. For years that first step was BM25 — sparse, keyword-based scoring. It's fast and strong, but it has a fatal blind spot: it matches on words, not meaning. When the question and the answer passage phrase things differently — the vocabulary mismatch problem — BM25 misses.

DPR replaces keyword matching with dense matching. It learns to map both questions and passages into the same vector space so that a question sits close to the passages that answer it, regardless of shared words. "Tallest mountain" and "Everest" land near each other because they mean related things. Retrieval becomes: encode the question, find the nearest passage vectors. Simple — the work is in learning that space, and that's where the paper's cleverness lives.

DPR uses two separate BERT encoders: one turns a question into a vector, the other turns a passage into a vector. Similarity is just their dot product. This dual-encoder (or bi-encoder) design has a decisive practical property: because a passage's vector doesn't depend on the question, you can encode all passages once, offline, store them in a vector index, and at query time only encode the question and do a fast nearest-neighbor search.

That independence is what makes dense retrieval scale to millions of passages — it's the same reason Sentence-BERT uses a bi-encoder rather than scoring every pair jointly. The cost is that question and passage never "see" each other during encoding, so the model must pack all the matching signal into the vectors themselves. Training has to be good enough to make that work — which brings us to the hard part: where do the negative examples come from?

To learn a good space, the model needs to see not just which passage is right for a question (the positive) but many that are wrong (negatives) — and it must push those away. Mining hard negatives for every question is expensive. DPR's trick, in-batch negatives, gets them almost for free.

The insight: in a batch of B pairs, every passage was going to be encoded anyway. So compute the full B×B similarity matrix. For question i, passage i is the positive; the other B−1 passages — the positives of the other questions — are perfectly good negatives, because a passage answering a different question is (almost always) wrong for this one. Training is then a simple in-batch classification: for each row, softmax over the B passages and push probability onto the diagonal. One batch yields B positives and B×(B−1) negatives at nearly zero extra cost.

Let a batch have B pairs, with question vectors qᵢ and passage vectors pⱼ. The similarity matrix and the negative count are:

A batch of 128 pairs yields 128 positives and 128·127 = 16,256 negatives — from encoding just 128 questions and 128 passages. The negatives scale as B², the encoding cost as B.

The loss is the negative log-likelihood of the positive under a softmax over each row — a contrastive objective that pulls the positive up and pushes the in-batch negatives down:

Maximizing the positive's share of the row's softmax makes the diagonal dominate — so at inference the top passage by dot product is the right one. Bigger batches mean more negatives per positive, which is why DPR benefits from large batch sizes. The runnable version below builds the B×B matrix and shows the diagonal winning.

Dense Passage Retrieval (Karpukhin, Oğuz, Min, Lewis, Wu, Edunov, Chen and Yih, EMNLP 2020) is a recipe more than an architecture: two off-the-shelf BERT encoders, a fixed passage format, a particular choice of negatives and a vector index. Each choice was tested, and the steps below are the configuration that won.

1. Cut Wikipedia into passages. The English Wikipedia dump of 20 December 2018, stripped of tables, infoboxes, lists and disambiguation pages, is split into disjoint 100-word blocks: 21,015,324 passages. Each passage is prefixed with its article title and a [SEP] token.
2. Encode with two BERTs. A question encoder E_Q and a passage encoder E_P, each an independent BERT-base (uncased), output the [CLS] vector, 768 numbers. Similarity is the plain dot product E_Q(q) · E_P(p).
3. Find positives. For Natural Questions and SQuAD, the passage that matches the gold context is the positive. TriviaQA, WebQuestions and CuratedTREC have no gold passage, so the highest-ranked BM25 passage containing the answer is used instead, and questions with no such passage in the top 100 are dropped.
4. Assemble negatives. Each question gets one BM25 hard negative: a top-ranked BM25 passage that does not contain the answer but matches most of the question's tokens. In a batch of B questions, every question is scored against all B gold passages and all B hard negatives, so it sees B − 1 in-batch negatives plus B hard ones.
5. Optimise. Batch size 128, Adam at a learning rate of 10^⁻⁵ with linear warm-up and decay, dropout 0.1, up to 40 epochs on the large datasets and 100 on the small ones (WebQuestions and CuratedTREC). A multi-dataset model is trained on the union of every training set except SQuAD.
6. Index offline. Encode all 21 million passages once — the paper reports about 8.8 hours on 8 GPUs — and load the vectors into FAISS for maximum-inner-product search.
7. Retrieve and read. At query time encode only the question, take the top k passages, and hand them to a BERT-based reader that scores each passage and extracts an answer span. With FAISS the paper retrieved the top 100 passages for 995 questions per second, against 23.7 questions per second per CPU thread for a Lucene BM25 index.
8. Optionally fuse with BM25. The hybrid takes the top 2,000 passages from each system and re-ranks the union by BM25(q, p) + λ · sim(q, p), with λ = 1.1.

The negative-type ablation, run on the Natural Questions development set, is what fixed steps 4 and 5. Random passages, BM25 passages and other questions' gold passages were each tried as negatives without in-batch reuse and ended up close to one another; reusing the batch's gold passages was clearly better and improved as the batch grew; and one BM25 hard negative per question on top added the largest single gain. A second BM25 negative did not help further.

• EQ, EP — two separate BERT-base encoders, each returning its [CLS] vector of 768 numbers.
• pi^+ — question i's positive: the gold context where one exists, otherwise the top BM25 passage that contains the answer.
• pj^− — every other passage scored in the batch: the other B − 1 gold passages and all B BM25 negatives.
• S — the whole batch's scores in one matrix product of question vectors against passage vectors.
• λ — the weight that puts the dense score on a scale comparable with BM25 in the hybrid.

At the extremes: with B = 1 and no hard negative the softmax has a single entry, the loss is always zero and nothing is learned; with in-batch negatives only, the model learns to reject passages about unrelated questions but rarely meets a plausible wrong one. That is why adding one BM25 negative per question moved top-20 accuracy on the Natural Questions development set from 73.0 to 78.0 at batch size 128. The paper also tried cosine and Euclidean similarity and a triplet loss: the dot product and Euclidean distance performed about the same, both beat cosine, and the triplet loss changed little.

Retrieval is measured as top-k accuracy: the share of test questions for which at least one of the top k retrieved passages contains the answer. The baseline is Lucene BM25 over the same 21 million passages, and the rows below are the single-dataset DPR models.

On four datasets the gain at top-20 is 9 to 19 points. SQuAD is the negative result, and the paper explains it: annotators wrote the questions while looking at the passage, so question and passage share many words, which favours BM25; and the training questions come from only about 500 Wikipedia articles, so the encoder learns from a narrow slice. Fusing the two scores recovered it, with the hybrid reaching 71.5 at top-20.

Two further results made dense retrieval practical. Trained on only 1,000 Natural Questions examples, DPR already beat BM25 at top-20, so the method did not need massive supervision. And better passages carried through to answers: with the paper's extractive reader, exact match on Natural Questions was 41.5 with DPR's passages against 32.6 with BM25's, ahead of the earlier dense-retrieval systems ORQA (33.3) and REALM (40.4), both of which needed extra pretraining.

In-batch negatives are cheap and easy: a random passage is almost always obviously wrong, so the model learns to reject the obvious and not to discriminate among plausible candidates. The paper addresses this by mixing in hard negatives — high-scoring wrong passages retrieved by a keyword system — and the improvement from doing so is large. The general lesson is worth more than the detail: a contrastive model is only as good as its hardest negatives, and free negatives are free because they are easy.

The other limitation is that the relation is trained, and therefore tied to the distribution it was trained on. Dense retrievers of this kind were repeatedly found to generalise worse than keyword scoring on out-of-domain collections — which is an unfashionable result that has held up, and the reason hybrid retrieval (dense plus keyword, scores combined) is the default production answer rather than a compromise.

How it held up: it established the dual-encoder-plus-vector-index architecture that every retrieval system on this site uses, and it made "train the retriever" an obvious step rather than an exotic one. Its own weakness — that compressing a passage into one vector loses term-level detail — is what rung 7 attacks directly.

DPR is a foundation stone of modern retrieval-augmented generation. The pattern it standardized — dual-encoder embeddings, dot-product matching, a vector index for fast search — is exactly how "chat with your documents" systems retrieve today. Every embedding-based retriever and vector database owes something to the recipe DPR made work, and in-batch negatives became a default technique for training contrastive embedding models well beyond QA.

It also marked a boundary later work pushed on. A single dense vector per passage can blur fine detail, which motivated late-interaction methods like ColBERT that keep per-token vectors, and hybrid systems that combine dense retrieval with BM25 to recover exact-match strengths. But the core lesson endures: match on meaning, encode passages offline, and train with cheap in-batch negatives. DPR turned retrieval from a keyword-matching problem into a representation-learning one — and that shift underlies the whole retrieval stack we build on now.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

The longest ladder here, and the most linear: nine papers that carry one idea — text as geometry — from a 2013 word embedding to a graph-structured retrieval system.

1. 1Word2Vec2013
2. 2GloVe2014
3. 3Sentence-BERT2019
4. 4Product Quantization2011
5. 5HNSW2016
6. 6Dense Passage Retrieval2020
7. 7ColBERT2020
8. 8Retrieval-Augmented Generation2020
9. 9GraphRAG2024

All 110 papers, in 19 ladders →