The whole design, in writing
Learn AI system design by building a reranking service — the precision tier of two-stage retrieval for search and RAG — step by step. An interactive guide covering why vector similarity alone gets the top result wrong, the retrieve-wide-then-rerank-narrow pattern, cross-encoder scoring, the top-K latency/cost tradeoff, serving and caching, hybrid retrieval fusion, and the unhappy paths (recall gates precision, latency budget, domain tuning).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a reranker for?
Vector search is great at finding relevant documents but mediocre at ordering them — the genuinely best passage often lands at rank 5, not rank 1. For search that's a worse experience; for RAG it's worse still, because the LLM only reads the top few passages, so a bad ordering feeds it weak context and the answer suffers.
A reranking service is the precision tier of two-stage retrieval: a fast first stage retrieves a wide candidate set (high recall), then a slower, more accurate reranker reorders the top of it (high precision), so the best passages rise to the top. It's the cheapest big win in retrieval quality — and the key to good RAG context.
What the new pieces do
- Clientclient
- A search or RAG request that wants the *best* passages at the very top — not just a bag of roughly-relevant ones.
Step 1 · Good recall, rough order
Why vector search isn't enough
Embedding-based (bi-encoder) retrieval embeds the query and every document separately, then compares vectors. That's fast and finds relevant docs — so what's the weakness that makes the top result often wrong?
Bi-encoder vector search finds relevant docs fast. Why is its top-1 often not the best?
A bi-encoder turns the query and each document into vectors separately and compares them, which is fast and scalable (you can precompute doc vectors) but never lets the model look at the query and a specific document together, so it misses fine-grained relevance signals. It reliably finds relevant candidates but orders them imperfectly — the best one is often not ranked first.
Dimension isn't the issue. The limitation is that query and document are encoded independently, so subtle query-document interactions that decide true relevance are never modeled — great recall, rough precision.
Bi-encoders scale extremely well (that's their point — precompute doc vectors, fast nearest-neighbor). Their weakness is ordering precision, because they never score the query and a document jointly.
A bi-encoder embeds the query and each document independently and compares vectors. That's why it scales (doc vectors are precomputed, search is fast nearest-neighbor) — but it never looks at the query and a specific document together, so it misses the fine-grained interactions that decide true relevance. Result: excellent recall (it finds the relevant docs) but imperfect ordering (the best one often isn't first).
Why this piece earns its place
The gap between finding and ordering survives in production because nothing reports it. A vector search hands back a full result set with plausible scores whether the best passage sits at position one or position sixty — no error, no latency spike, no dashboard that moves. The score is no help either: cosine similarity is comparable within one query and close to meaningless across queries, because the number depends on where that particular query happens to land in the embedding space, so there is no distribution to threshold on. It also decays without anyone touching the code. Every document you add is another chance for something plausible to sit above the right answer, so the order a query got at launch is not the order it gets a year and a corpus later, and the release notes for that regression are empty. The structural problem is the worse one. Document vectors are written at index time, so first-stage ordering is a property of how the corpus was built, not a knob available at query time — improving it means re-embedding everything you have. That asymmetry, not the accuracy gap on its own, is what makes a second stage worth paying for.
Step 2 · Two stages
Retrieve wide, then rerank narrow
You want the accuracy of scoring the query against each document jointly, but that model is far too slow to run over a whole corpus. How do you get both scale and precision?
A joint query-document scorer is accurate but too slow for the whole corpus. How do you use it?
Scoring the query jointly against millions of documents is computationally impossible at query time — that's exactly why you can't use it as the first stage. You run it only on a small candidate set.
That leaves the top-result-often-wrong problem, which especially hurts RAG. The fix isn't to abandon precision but to apply the expensive, accurate model to only the few candidates that matter.
Stage one uses the cheap bi-encoder/keyword retrieval to pull, say, the top 100–200 candidates (high recall, low cost). Stage two runs the expensive joint scorer on only those candidates to reorder them precisely. You get corpus-scale recall AND top-of-list precision, paying the expensive model only on a small set.
Split into two stages. Stage 1 — retrieve wide: use fast, cheap retrieval (vector ANN + keyword) to pull a broad candidate set (e.g. top 100–200) with high recall. Stage 2 — rerank narrow: run the expensive, accurate joint scorer on only those candidates to reorder them with high precision, returning the sharpened top-N. You get corpus-scale recall and top-of-list precision, paying the costly model on a small set instead of the whole corpus.
Why this piece earns its place
Splitting the pipeline creates an interface, and two properties of it are worth settling before anything is built. The first is the unit. If the index holds passages but the reranker is handed whole documents, the stages are counting different things, and the candidate count stops describing how much distinct material is on the table — the same number is a wide net when the hits come from different sources and a narrow one when most of them are cut out of the same manual. Pick the unit once, make the first stage deduplicate to it, or top-K stops meaning anything you can reason about. The second is the payload. A cross-encoder reads text, not identifiers, so either the first stage carries the text of every candidate in its response, or the second stage fetches it from the document store — a fan-out on the hot path that nobody costs at design time and that shows up first at load test. Both are workable once chosen. The reward for taking the seam seriously is that the first stage becomes swappable: add a keyword index, a recency filter, a second retriever, and nothing downstream is retrained, because the second stage wants only a query and some text.
- stage 1retrieve ~100+ (recall)
- stage 2rerank to top-N (precision)
- costexpensive model on few
What the new pieces do
- Retrieval APIgateway
- Runs the two-stage pipeline: retrieve a wide candidate set fast, then rerank the top of it for precision.
Step 3 · Cast a wide net
First-stage retrieval (recall)
The first stage's one job is to make sure the truly relevant documents are somewhere in the candidate set — because the reranker can only reorder what it's given. How do you maximize recall cheaply?
Run fast, high-recall retrieval: vector (ANN) search for semantic matches plus keyword/BM25 for exact terms, fused (hybrid retrieval) so you catch both meaning and specific words/names/IDs. Pull a generous candidate set (top 100–200) — wide enough that the best passages are almost certainly included, cheap enough to run over the whole index. This stage optimizes recall, not order; getting the ordering right is the reranker's job.
Why this piece earns its place
Fusing two result lists is where hybrid retrieval usually goes wrong. BM25 scores and cosine similarities do not share a scale — the first is an unbounded sum of corpus statistics, the second is bounded and query-dependent — so a weighted sum needs both normalized first, and the normalization is the unstable part: it is fitted to a corpus that keeps growing, and it drifts as the corpus does. Rank fusion sidesteps the problem by throwing the scores away and combining positions instead, which is why reciprocal rank fusion is the default rather than a clever arithmetic blend. The weighting that does earn its keep is which list to lean on, and that is a per-query judgement rather than a global constant — so it belongs in the request, set by a cheap classifier in front of the retriever, not frozen in the index config. Then two arithmetic traps. The two lists overlap, so each retriever has to be asked for more than the fused set you want, or the wide net is narrower than the config reads. And a metadata filter applied after the ANN search — tenant, permissions, recency — shrinks that set again, so a request can reach the reranker holding a handful of candidates while the config still says top 100–200. Filter inside the index, or over-fetch on purpose.
- vector + keywordhybrid
- top 100–200wide net
- recallbest doc must be present
What the new pieces do
- Retrieverservice
- First stage: fast, high-recall retrieval (vector ANN + keyword/BM25, fused) that casts a wide net for candidates.
- Indexdata
- The vector (and keyword) index the first stage queries to pull the top candidate documents cheaply.
Step 4 · Score them properly
The cross-encoder reranker (precision)
Now reorder the candidates by true relevance to the query. What model does this, and why is it accurate but expensive?
Use a cross-encoder: it takes the query and a candidate together as one input and outputs a precise relevance score, because the model attends over both jointly and captures the fine-grained interactions a bi-encoder can't. Run it on each of the ~100 candidates, then sort by score to get the final order. It's far more accurate than vector similarity — and far more expensive, since it's a full model pass per candidate (you can't precompute it, as the score depends on the specific query). That cost is exactly why it runs only on the small candidate set, not the corpus.
Why this piece earns its place
Two things about a cross-encoder only show up once it is serving traffic. The first is that the input window is a shared budget. Query and candidate are concatenated into one sequence with a fixed limit, so a long query — the user's whole question, plus whatever the product prepends to it — spends length the candidate then does not get, and what falls off the end is the tail of the candidate. Track the share of pairs that hit the limit; it is the one number that tells you whether the model is reading your candidates or the first half of them. The second is what a real score buys beyond sort order. Because a cross-encoder judges a pair rather than a point in space, its output can be calibrated against labelled data into a usable cutoff — return the candidates that clear the bar instead of always returning N, which for RAG is worth more than the ordering itself. But that calibration belongs to one checkpoint. A new model's scores sit on their own scale, so a threshold carried across a version bump either floods the prompt with weak passages or starves it. Version the cutoff with the model, and re-fit it before the swap rather than after.
- cross-encoderquery+doc jointly
- per candidatea model pass each
- sortby relevance score
What the new pieces do
- Rerankerservice
- Second stage: scores each (query, candidate) pair together with a slow but accurate cross-encoder, then sorts.
- Rerank Modeldata
- The cross-encoder that reads the query and a candidate jointly and outputs a precise relevance score.
Step 5 · How many, how fast
The top-K latency/cost tradeoff
Reranking cost is linear in the number of candidates — rerank 100 and it's 10× the cost/latency of reranking 10. Rerank too few and you might miss the best doc (it was at rank 60); rerank too many and you blow the latency budget. What's the knob?
Reranking cost scales with candidate count. How do you choose how many to rerank?
The number of candidates reranked (top-K) is the core dial: large enough that the best passage is almost always in the set (recall), small enough to fit the latency and cost budget (each is a cross-encoder pass). You tune it empirically and batch all K candidates through the GPU together for throughput.
Reranking everything the first stage returns can blow the latency budget if that set is large, with diminishing returns past a point. You pick a top-K that captures the best docs without over-paying.
Reranking too few defeats the purpose — if the best passage was at rank 30 in stage one, a reranker that only sees the top 2 can never surface it. You need enough candidates to include it.
Tune top-K — how many candidates you rerank — as the core dial. It must be large enough that the best passage is almost always within it (so the first stage's recall is realized) but small enough to fit the latency and cost budget, since each candidate is a full cross-encoder pass. Batch all K candidates through the GPU together for throughput, keep the reranker model warm, and pick K empirically (often tens to a couple hundred). This knob directly trades quality against speed and cost.
Why this piece earns its place
Linear in candidates is the billing model, not the latency model. The K pairs go through as one padded batch, so a request's wall time is set by the longest sequence in that batch rather than the average, and the batch dimension runs in parallel until the hardware is full. Two consequences follow, and they point at different knobs. Inside a request, one long candidate slows every other candidate beside it, so tail latency tracks the longest document you admit — which means a hard truncation length, or bucketing candidates by length, buys latency that trimming K does not. Across the fleet, the work is K model passes per query however fast any single pass is, so capacity is K multiplied by queries per second and raising K is a hardware decision written as a one-line config edit. That is the rule for which knob to reach for: when one request misses the budget, cap length first; when the fleet cannot hold the rate, a smaller K is the honest fix, and it is the one that costs quality. Under light traffic K looks free and is quietly spending money; once the GPUs saturate, the queue forms and it starts spending latency too. Load-test at the traffic you actually have.
- top-Kthe core dial
- linear costa pass per candidate
- batch KGPU throughput
What the new pieces do
- Top Resultsdata
- The final, sharply-ordered top-N passages returned to search, or handed to the LLM as RAG context.
Step 6 · Serve it in the path
Serving, caching & fusion
The reranker lives on the query hot path (search or RAG), so it must be fast and cheap to operate. How do you serve it well?
Run reranking as a service on the retrieval path: batch candidates, keep the model warm/autoscaled (GPU), and cache reranked orders for repeated queries so hot searches skip the cross-encoder entirely. Fuse first-stage signals (vector + keyword scores) before reranking to feed the reranker a strong candidate set, and optionally blend the rerank score with other signals (recency, popularity, business rules) for the final order. The output feeds search results or becomes the top passages for RAG.
Why this piece earns its place
The cache is the piece that looks trivial and leaks. A reranked order is reusable only if everything that produced it is in the key: the normalized query, the filter set, the tenant, K, and the model version. Leave the tenant out and one customer is served an order computed over another's documents. Leave the model version out and a deploy quietly serves last week's ranking until entries age out. And these entries do not go stale when the query changes — they go stale when the corpus does, an event the cache never sees, so a TTL matched to how fast the index moves is the honest answer rather than real invalidation. Then size it for the consumer you actually have, because hit rate splits hard between them and the split is not obvious from the outside. RAG is the one to check first: its questions arrive as whole sentences, phrased differently every time, so a cache that pays for itself in front of a search box earns close to nothing on that path, and the same money buys more as batch headroom and a warmer pool. Two consumers, one service, and only one of them should be capacity-planned around cache hits.
- cacherepeated queries
- batch + warmaffordable serving
- fuserelevance + rules
What the new pieces do
- Cachecache
- Caches reranked results for repeated queries so hot searches skip the expensive cross-encoder pass.
Step 7 · Recall gates precision
The reranker can only sort what it's given
It's tempting to think a great reranker fixes retrieval. It can't. What's the hard limit, and what does it mean for how you invest?
A reranker only reorders the candidates it receives — it cannot conjure a relevant document the first stage failed to retrieve. So if the right passage isn't in the candidate set, no reranker can surface it: recall gates precision. This means the first stage's job (don't miss the relevant docs) and the second stage's job (sort them to the top) are complementary and both essential. Invest in first-stage recall (hybrid retrieval, enough candidates) and second-stage precision (a good reranker, right top-K) — neither substitutes for the other.
Why this piece earns its place
This is measurable, and the measurement is the part worth carrying into the interview. Run the first stage alone over a judged query set and compute recall at exactly the K you rerank — not at the candidate count, at K. That single number is the reranker's ceiling for top-one accuracy, and no second stage can beat it. Compute it before you build, because it prices the work: a ceiling far above what you ship today says the ordering is where your quality is hiding, while a ceiling sitting barely above it says every hour spent on the reranker is capped by the first stage until the first stage moves. The same instrument settles the arguments afterwards. A wrong result is either a recall miss or an ordering miss, and the output alone cannot tell you which, so log each result's rank in both the pre-rerank and post-rerank orders. Never in the list at all: widen K or fix the first stage, and reranker work is wasted. Deep in the list before and still deep after: the reranker is the problem. Two repairs, opposite directions, and one log line keeps you off the wrong one.
- rerankerreorders only
- recallgates precision
- both stagesessential
Step 8 · The sharp edges
Latency, domain fit & cost
Real reranking must fit a latency budget on the hot path, generalize (or not) to your domain, and stay cost-effective while genuinely improving quality.
Hold the latency budget with a tuned top-K, GPU batching, a warm autoscaled reranker, and caching — accepting a smaller K under tight budgets. Watch domain fit: off-the-shelf rerankers may underperform on specialized domains, so evaluate on your data and fine-tune if needed. Manage cost (linear in K) with caching, right-sized models, and reranking only when it helps (skip for trivial queries). And measure: confirm reranking actually lifts your retrieval metrics (nDCG, top-k accuracy) and downstream RAG answer quality — it usually does, which is why it's the highest-ROI retrieval upgrade.
Why this piece earns its place
Fine-tuning on your domain is the right instinct, and labels are what stop it. You have no relevance judgments, but you do have clicks — and citations from the RAG side — so those become the training data, and they were produced by the order you already showed. People click what is near the top. Train on that and you teach the reranker to reproduce your current ranking: it scores well offline and moves nothing in production. Usable labels mean breaking the loop on purpose — interleave or randomize a slice of traffic, or buy human judgments over queries you genuinely receive rather than queries someone imagined. The second-order point is which artifact deserves the care. Everything else here is replaceable: the checkpoint, the embedding model, the fusion weighting, K. Each of those swaps is either a gain or a regression, and nothing on the serving path can tell the two apart — only the same judged queries, run before and after. So build that set once, version it apart from the corpus, and keep it frozen between releases; otherwise a bad week and a changed set look identical.
You did it
You just designed a reranking service.
Everything you assembled, in order
- Vector (bi-encoder) search has great recall but rough ordering — the best passage often isn't first.
- Two-stage retrieval: retrieve wide (fast, recall) then rerank narrow (slow, precise).
- First stage = hybrid (vector + keyword) high-recall retrieval pulling a wide candidate set.
- Second stage = a cross-encoder scoring each (query, candidate) jointly for precise relevance.
- Top-K is the core dial: enough candidates to contain the best doc, few enough for the latency/cost budget.
- Serve it batched, warm and cached on the query path; fuse signals; feed the top passages to RAG.
- Recall gates precision — a reranker only reorders what retrieval found, so both stages are essential.