The big idea

What is a reranker for?

Vector search is great at finding relevant documents but mediocre at ordering them — the genuinely best passage often lands at rank 5, not rank 1. For search that's a worse experience; for RAG it's worse still, because the LLM only reads the top few passages, so a bad ordering feeds it weak context and the answer suffers.

A reranking service is the precision tier of two-stage retrieval: a fast first stage retrieves a wide candidate set (high recall), then a slower, more accurate reranker reorders the top of it (high precision), so the best passages rise to the top. It's the cheapest big win in retrieval quality — and the key to good RAG context.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end kill the reranker and break first-stage retrieval to see precision vs the recall that gates it. Hit Begin.

Step 1 · Good recall, rough order

Why vector search isn't enough

Embedding-based (bi-encoder) retrieval embeds the query and every document separately, then compares vectors. That's fast and finds relevant docs — so what's the weakness that makes the top result often wrong?

Design decision: Bi-encoder vector search finds relevant docs fast. Why is its top-1 often not the best?

The call: Because vectors are too small. — Dimension isn't the issue. The limitation is that query and document are encoded independently, so subtle query-document interactions that decide true relevance are never modeled — great recall, rough precision.

A bi-encoder embeds the query and each document independently and compares vectors. That's why it scales (doc vectors are precomputed, search is fast nearest-neighbor) — but it never looks at the query and a specific document together, so it misses the fine-grained interactions that decide true relevance. Result: excellent recall (it finds the relevant docs) but imperfect ordering (the best one often isn't first).

Bi-encoder = scalable but coarse: Independent encoding is what makes vector search fast enough to run over millions of docs — and also what caps its precision. To judge "is this document the best answer to this query?", you need a model that reads both together. That model is too slow to run over everything, which sets up two-stage retrieval.

Step 2 · Two stages

Retrieve wide, then rerank narrow

You want the accuracy of scoring the query against each document jointly, but that model is far too slow to run over a whole corpus. How do you get both scale and precision?

Design decision: A joint query-document scorer is accurate but too slow for the whole corpus. How do you use it?

The call: Two stages: fast retrieval gets a wide candidate set (recall), then the accurate model reranks just those few (precision). — Stage one uses the cheap bi-encoder/keyword retrieval to pull, say, the top 100–200 candidates (high recall, low cost). Stage two runs the expensive joint scorer on only those candidates to reorder them precisely. You get corpus-scale recall AND top-of-list precision, paying the expensive model only on a small set.

Split into two stages. Stage 1 — retrieve wide: use fast, cheap retrieval (vector ANN + keyword) to pull a broad candidate set (e.g. top 100–200) with high recall. Stage 2 — rerank narrow: run the expensive, accurate joint scorer on only those candidates to reorder them with high precision, returning the sharpened top-N. You get corpus-scale recall and top-of-list precision, paying the costly model on a small set instead of the whole corpus.

Recall stage + precision stage: The pattern that powers modern search and RAG: a cheap high-recall retriever narrows millions to hundreds; an expensive high-precision reranker narrows hundreds to the best few. Each stage does what it's good at, and the reranker only ever sees a small, affordable candidate set.

Step 3 · Cast a wide net

First-stage retrieval (recall)

The first stage's one job is to make sure the truly relevant documents are somewhere in the candidate set — because the reranker can only reorder what it's given. How do you maximize recall cheaply?

Run fast, high-recall retrieval: vector (ANN) search for semantic matches plus keyword/BM25 for exact terms, fused (hybrid retrieval) so you catch both meaning and specific words/names/IDs. Pull a generous candidate set (top 100–200) — wide enough that the best passages are almost certainly included, cheap enough to run over the whole index. This stage optimizes recall, not order; getting the ordering right is the reranker's job.

Hybrid retrieval: Vector search catches semantic similarity; keyword search catches exact matches vector search fumbles (rare terms, codes, names). Fusing them (e.g. reciprocal rank fusion) maximizes recall so the candidate set almost always contains the right answer — which is the precondition for the reranker to succeed.

Step 4 · Score them properly

The cross-encoder reranker (precision)

Now reorder the candidates by true relevance to the query. What model does this, and why is it accurate but expensive?

Use a cross-encoder: it takes the query and a candidate together as one input and outputs a precise relevance score, because the model attends over both jointly and captures the fine-grained interactions a bi-encoder can't. Run it on each of the ~100 candidates, then sort by score to get the final order. It's far more accurate than vector similarity — and far more expensive, since it's a full model pass per candidate (you can't precompute it, as the score depends on the specific query). That cost is exactly why it runs only on the small candidate set, not the corpus.

Cross-encoder vs bi-encoder: A bi-encoder encodes query and doc separately (fast, precomputable, coarse). A cross-encoder encodes them together (slow, per-pair, precise). The reranker is the cross-encoder applied to the shortlist — the accuracy you couldn't afford over everything, made affordable by only scoring a few.

Step 5 · How many, how fast

The top-K latency/cost tradeoff

Reranking cost is linear in the number of candidates — rerank 100 and it's 10× the cost/latency of reranking 10. Rerank too few and you might miss the best doc (it was at rank 60); rerank too many and you blow the latency budget. What's the knob?

Design decision: Reranking cost scales with candidate count. How do you choose how many to rerank?

The call: Always rerank the entire candidate set from stage one. — Reranking everything the first stage returns can blow the latency budget if that set is large, with diminishing returns past a point. You pick a top-K that captures the best docs without over-paying.

Tune top-K — how many candidates you rerank — as the core dial. It must be large enough that the best passage is almost always within it (so the first stage's recall is realized) but small enough to fit the latency and cost budget, since each candidate is a full cross-encoder pass. Batch all K candidates through the GPU together for throughput, keep the reranker model warm, and pick K empirically (often tens to a couple hundred). This knob directly trades quality against speed and cost.

K balances recall realized vs cost: Rerank-K sits between the stages: too small and you can't recover a great doc the first stage ranked low; too large and you pay linear cost for diminishing gains. Batching the K (query, doc) pairs on the GPU keeps the expensive stage fast enough for interactive latency.

Step 6 · Serve it in the path

Serving, caching & fusion

The reranker lives on the query hot path (search or RAG), so it must be fast and cheap to operate. How do you serve it well?

Run reranking as a service on the retrieval path: batch candidates, keep the model warm/autoscaled (GPU), and cache reranked orders for repeated queries so hot searches skip the cross-encoder entirely. Fuse first-stage signals (vector + keyword scores) before reranking to feed the reranker a strong candidate set, and optionally blend the rerank score with other signals (recency, popularity, business rules) for the final order. The output feeds search results or becomes the top passages for RAG.

A precision layer you can drop in: The reranker is a modular stage you insert between retrieval and consumption. Caching (queries repeat), batching (K pairs), and warm autoscaled serving keep it affordable; signal fusion lets relevance combine with product priorities. For RAG, it decides which passages the LLM actually reads.

Step 7 · Recall gates precision

The reranker can only sort what it's given

It's tempting to think a great reranker fixes retrieval. It can't. What's the hard limit, and what does it mean for how you invest?

A reranker only reorders the candidates it receives — it cannot conjure a relevant document the first stage failed to retrieve. So if the right passage isn't in the candidate set, no reranker can surface it: recall gates precision. This means the first stage's job (don't miss the relevant docs) and the second stage's job (sort them to the top) are complementary and both essential. Invest in first-stage recall (hybrid retrieval, enough candidates) and second-stage precision (a good reranker, right top-K) — neither substitutes for the other.

Two jobs, both required: Stage one: find the relevant docs (recall). Stage two: rank them correctly (precision). A perfect reranker over a candidate set missing the answer still fails; perfect recall in a jumbled order still buries the best doc. The system is only as good as the weaker stage.

Step 8 · The sharp edges

Latency, domain fit & cost

Real reranking must fit a latency budget on the hot path, generalize (or not) to your domain, and stay cost-effective while genuinely improving quality.

Hold the latency budget with a tuned top-K, GPU batching, a warm autoscaled reranker, and caching — accepting a smaller K under tight budgets. Watch domain fit: off-the-shelf rerankers may underperform on specialized domains, so evaluate on your data and fine-tune if needed. Manage cost (linear in K) with caching, right-sized models, and reranking only when it helps (skip for trivial queries). And measure: confirm reranking actually lifts your retrieval metrics (nDCG, top-k accuracy) and downstream RAG answer quality — it usually does, which is why it's the highest-ROI retrieval upgrade.

Design for the unhappy path: Latency → tune K, batch, cache. Domain → evaluate + fine-tune. Cost → cache + right-size + skip when pointless. Always measure the lift. Reranking is famously the cheapest large gain in retrieval/RAG quality — but only if recall is solid and you've tuned K to your budget.

You did it

You just designed a reranking service.

• V — e
• T — w
• F — i
• S — e
• T — o
• S — e
• R — e