Reranking Retrieved Passages

A vector database can compare a query embedding against a million passage embeddings in milliseconds because it uses a cheap similarity measure — dot product or cosine similarity between two fixed vectors, computed independently of each other. That speed comes at a cost: the passage's embedding was computed without ever seeing the query, so the comparison can only be as good as squeezing all of a passage's meaning into one fixed vector allows. A reranker fixes this by looking at the query and each candidate passage together, jointly, which is far more accurate but far too slow to run over an entire corpus — so it's applied only to the shortlist that fast retrieval already narrowed down.

Why a two-stage pipeline

Retrieval and reranking trade off speed against accuracy at different scales. The first-stage retriever (BM25, embeddings, or a hybrid of both) has to consider every passage in the corpus, so it must be cheap per comparison — commonly returning the top 50 or 100 candidates out of possibly millions. The reranker then runs a cross-encoder — a model that takes the query and one passage as a single joint input and outputs a relevance score — over just those 50–100 candidates, reordering them by a judgment that accounts for how the query and passage interact rather than comparing two vectors computed in isolation. Running a cross-encoder over the whole corpus would be far too slow for interactive use; running only the fast first-stage retriever without reranking leaves relevance on the table that a joint model would have caught.

Worked example: reranking five earnings-call candidates

A query — "did management mention supply chain constraints easing?" — returns five candidate passages from first-stage hybrid retrieval, already ranked by their combined BM25/embedding score: passage 1 mentions "supply chain" three times but is actually about hiring; passage 2 is the CFO saying constraints "have started to normalize"; passage 3 mentions "chain of command" (a keyword collision with "chain"); passage 4 mentions cost pressure generally without saying whether it's easing; passage 5 is an analyst's question repeating the query's own wording without an answer attached. First-stage retrieval, driven partly by keyword overlap, might rank passage 1 or 5 highly because of surface term overlap. A cross-encoder reranker, reading the query and each passage jointly, recognizes that passage 2 is the only one that actually answers the question and pushes it to the top — a reordering that a query-independent embedding comparison could easily miss, since "have started to normalize" doesn't lexically or even semantically resemble "easing" as strongly as passage 1's repeated "supply chain" phrase does on the surface.

What this means in practice

Reranking is usually the single highest-leverage addition to a retrieval pipeline once basic hybrid search is in place, because it directly attacks the gap between "roughly related" and "actually answers the question" — the gap that matters most once the passages reach an LLM that will summarize or quote them. It's also where citation quality is won or lost: if the reranker doesn't put the truly relevant passage at position 1, the LLM downstream may cite a plausible-looking but wrong passage with full confidence.