Cross-encoder vs bi-encoder: retrieve fast, or rank well?

A bi-encoder passes the query and each document through the model separately, turning each into its own vector; relevance is just the similarity between two independent vectors. Because a document’s vector doesn’t depend on the query, you can precompute all document vectors offline and, at query time, encode only the query and search the index. A cross-encoder feeds the query and a document through the model together, in one pass, and outputs a single relevance score — the two texts attend to each other directly, which is far more accurate, but the score exists only for that specific pair and can’t be precomputed.

You don’t choose — you stage them. The standard production pattern is retrieve-then-rerank: a bi-encoder retrieves a broad shortlist (say the top 100) from the whole corpus in milliseconds, then a cross-encoder rescoring reorders just those 100 into the final top 10. The bi-encoder does the impossible-at-scale part (scanning everything); the cross-encoder does the impossible-at-scale-but-cheap-on-100 part (reading each candidate carefully). You get the bi-encoder’s reach and most of the cross-encoder’s accuracy, for the cost of ~100 extra model passes per query.

If cross-encoders are more accurate, why not use one for the whole search? Arithmetic. A cross-encoder produces a score only for a specific query–document pair, so ranking a corpus of one million documents against one query would require one million model forward passes — per query. That’s seconds to minutes of GPU time for a single search, which no interactive system can afford. The bi-encoder sidesteps this entirely: every document is encoded once, offline, into a vector; at query time you encode just the query and let an approximate-nearest-neighbor index find the closest vectors in milliseconds, never touching the model per document.

The cross-encoder’s accuracy comes from the same thing that makes it unscalable — the query and document attending to each other inside the model. You can’t precompute that interaction, because it doesn’t exist until the query arrives. So the two aren’t really competitors: the bi-encoder makes the search possible, and the cross-encoder makes the top of it sharp.

You’re building search over 10 million support articles. Encoding all 10M with a cross-encoder per query is a non-starter — that’s 10M passes for one search. So you embed all 10M documents once with a bi-encoder and store the vectors in an ANN index. A query comes in; you encode it once and the index returns the top 100 candidates in a few milliseconds. Good — but the true best answer might be sitting at rank 40, because separate encoding missed a subtle interaction.

Now a cross-encoder rescoring reads the query together with each of those 100 candidates — 100 model passes, entirely affordable — and reorders them, pulling that rank-40 answer up to the top 3 where the user (or an LLM building an answer) will actually see it. Same two models, staged: retrieve broad and fast, then rerank narrow and sharp.