Reranking
A second, slower model re-scores your top search results and reorders them, so the genuinely best chunk lands first.
See it
What it is
Retrieval gets you a rough pile of candidates fast. Reranking is the second pass that reads each candidate against the actual query and reorders them. The speed difference is structural: a bi-encoder embeds query and document separately, so it can precompute millions of vectors, while a cross-encoder feeds the query and one document through the model together and scores the pair. Far more accurate, far too slow to run over a whole corpus.
So the standard shape is retrieve wide, rerank narrow: pull 50 to 100 chunks with vector or hybrid search, rerank them, keep the top 5 for the prompt. Hosted rerankers (Cohere Rerank, Voyage) and open models (bge-reranker, Jina) all take the same query-plus-documents shape. It is usually the highest-return change you can make to a mediocre RAG system, because the winning chunk was often already retrieved and just buried under near-misses.
Gotcha: reranking is a precision tool and cannot rescue bad recall. If the answer never made it into the candidate set, no amount of reordering conjures it. Second gotcha: cost and latency scale with candidate count, since every candidate is a separate forward pass. Reranking 200 chunks per query feels great in a demo and reads very differently on the invoice.
Ask AI for it
Add a reranking stage between retrieval and prompt assembly. Widen the retriever to return the top 50 candidate chunks, then pass the raw query plus those chunks to a cross-encoder reranker, keep the 5 highest-scoring chunks, and drop anything below a configurable relevance threshold even if that leaves fewer than 5. Do not hard-code the number: reranker scores are not comparable across models, so calibrate the threshold on a labelled set of query and chunk relevance judgements for the reranker you actually picked, report precision, recall and how often the query returns nothing at each candidate value, and pick the default from that table. Preserve the original chunk ids and source metadata through the rerank so citations still resolve, cache rerank results per query hash, and log both the pre-rerank and post-rerank order so the reordering is measurable.