Reranking
A second, smarter pass that reorders retrieved candidates before they reach the prompt.
First-stage retrieval optimizes recall over millions of documents; reranking optimizes precision over the top ~100. A cross-encoder reads query and document together — far more accurate than comparing separate embeddings, far too slow to run over everything. The retrieve-then-rerank cascade is the standard two-stage design, and often the single biggest RAG quality upgrade.
Worked example: retrieve the top 100 candidates cheaply with a bi-encoder, then a cross-encoder reads each query-document pair jointly and re-scores them to pick the best 5 — precision where it matters, speed where it does not. Gotcha: the reranker is far slower per item, so run it only on a shortlist; rerank too many and latency spikes, too few and the right answer never reaches it.