DPR,
explained.

Open-domain question answering starts with a retrieval step: pull a handful of relevant passages from millions, then read them for the answer. For years that first step was BM25 — sparse, keyword-based scoring. It's fast and strong, but it has a fatal blind spot: it matches on words, not meaning. When the question and the answer passage phrase things differently — the vocabulary mismatch problem — BM25 misses.

DPR replaces keyword matching with dense matching. It learns to map both questions and passages into the same vector space so that a question sits close to the passages that answer it, regardless of shared words. "Tallest mountain" and "Everest" land near each other because they mean related things. Retrieval becomes: encode the question, find the nearest passage vectors. Simple — the work is in learning that space, and that's where the paper's cleverness lives.

DPR uses two separate BERT encoders: one turns a question into a vector, the other turns a passage into a vector. Similarity is just their dot product. This dual-encoder (or bi-encoder) design has a decisive practical property: because a passage's vector doesn't depend on the question, you can encode all passages once, offline, store them in a vector index, and at query time only encode the question and do a fast nearest-neighbor search.

That independence is what makes dense retrieval scale to millions of passages — it's the same reason Sentence-BERT uses a bi-encoder rather than scoring every pair jointly. The cost is that question and passage never "see" each other during encoding, so the model must pack all the matching signal into the vectors themselves. Training has to be good enough to make that work — which brings us to the hard part: where do the negative examples come from?

To learn a good space, the model needs to see not just which passage is right for a question (the positive) but many that are wrong (negatives) — and it must push those away. Mining hard negatives for every question is expensive. DPR's trick, in-batch negatives, gets them almost for free.

The insight: in a batch of B pairs, every passage was going to be encoded anyway. So compute the full B×B similarity matrix. For question i, passage i is the positive; the other B−1 passages — the positives of the other questions — are perfectly good negatives, because a passage answering a different question is (almost always) wrong for this one. Training is then a simple in-batch classification: for each row, softmax over the B passages and push probability onto the diagonal. One batch yields B positives and B×(B−1) negatives at nearly zero extra cost.

Let a batch have B pairs, with question vectors qᵢ and passage vectors pⱼ. The similarity matrix and the negative count are:

A batch of 128 pairs yields 128 positives and 128·127 = 16,256 negatives — from encoding just 128 questions and 128 passages. The negatives scale as B², the encoding cost as B.

The loss is the negative log-likelihood of the positive under a softmax over each row — a contrastive objective that pulls the positive up and pushes the in-batch negatives down:

Maximizing the positive's share of the row's softmax makes the diagonal dominate — so at inference the top passage by dot product is the right one. Bigger batches mean more negatives per positive, which is why DPR benefits from large batch sizes. The runnable version below builds the B×B matrix and shows the diagonal winning.

DPR made a clean, influential case that learned dense retrieval beats decades-old sparse baselines:

The paper also showed that a modest number of labeled question-passage pairs was enough — you didn't need massive supervision to beat BM25. Combined with the cheap negatives, that made dense retrieval not just better but practical to train, which is a big part of why it caught on so fast.

DPR is a foundation stone of modern retrieval-augmented generation. The pattern it standardized — dual-encoder embeddings, dot-product matching, a vector index for fast search — is exactly how "chat with your documents" systems retrieve today. Every embedding-based retriever and vector database owes something to the recipe DPR made work, and in-batch negatives became a default technique for training contrastive embedding models well beyond QA.

It also marked a boundary later work pushed on. A single dense vector per passage can blur fine detail, which motivated late-interaction methods like ColBERT that keep per-token vectors, and hybrid systems that combine dense retrieval with BM25 to recover exact-match strengths. But the core lesson endures: match on meaning, encode passages offline, and train with cheap in-batch negatives. DPR turned retrieval from a keyword-matching problem into a representation-learning one — and that shift underlies the whole retrieval stack we build on now.