Why adding a reranker isn't improving your RAG results, and how to fix it
reranker makes no measurable difference to retrieval accuracy
Also appears as
- cross-encoder reranker returns the same ranking as vector search
- reranking adds latency without improving answer quality
- reranker scores don't correlate with actual relevance
Short answer
A reranker that shows no improvement usually means it is only reordering a candidate pool that was already too small, top-3 to top-5, to contain the correct answer, so there is nothing better to promote, or the wrong reranker model was chosen for the domain. Retrieve a wider candidate set of 20-50 before reranking, verify the reranker model actually outperforms your vector search on a labeled evaluation set, and budget the added latency deliberately rather than treating it as free.
Affects: Any RAG pipeline that has added a cross-encoder or LLM-based reranker on top of an existing vector search step.
Fastest path to a reranker that actually helps
- 1Widen the initial vector search to retrieve 20-50 candidates before reranking, not just the final 3-5 you intend to use.
- 2Confirm the reranker model is actually general-purpose or domain-appropriate; a generic cross-encoder trained on web search data may add little value on highly technical text.
- 3Measure reranker impact directly: compare recall@k and answer accuracy with reranking on versus off on the same evaluation set.
- 4Check that reranker scores are actually being used to reorder results, not just computed and logged without changing the final chunk selection.
- 5Account for the added latency in your latency budget, and reduce the candidate pool size if reranking latency is unacceptable rather than skipping reranking entirely.
How to confirm this is your problem
- Retrieval accuracy metrics are essentially unchanged before and after adding a reranker
- The reranked top result is identical or nearly identical to the original vector search top result
- Reranking adds noticeable latency to every query with no measurable accuracy benefit to show for it
- Reranker relevance scores look reasonable in isolation but don't change which chunks are ultimately used
- The reranker helps on general questions but not on domain-specific or technical queries
Root causes and fixes
Too few candidates are passed into the reranker for it to have any meaningful choice to make
If the vector search already narrows results down to the final top-3 or top-5 before the reranker runs, there is nothing left for the reranker to promote beyond what bi-encoder similarity already selected. A reranker's value comes from re-ordering a wider, noisier candidate pool where the correct answer is present but not necessarily ranked first.
Fix: Retrieve a wider candidate pool of 20-50 chunks with the initial vector search, then apply the reranker to that full pool and keep only the top 3-5 after reranking, giving the reranker actual room to correct the vector search's ordering mistakes.
The reranker model is not well suited to the domain or query style
Off-the-shelf cross-encoder rerankers are typically trained on general web search or open-domain QA data. On highly technical, legal, or domain-specific text with specialized vocabulary, a general-purpose reranker may not discriminate relevance any better than the original bi-encoder, since it was never exposed to enough in-domain examples to learn what matters in that specific context.
Fix: Evaluate two or three reranker models, including any domain-adapted options available, on a labeled in-domain evaluation set before committing to one, and consider fine-tuning a reranker on domain-specific query-passage pairs if a large enough labeled set is available.
Reranker scores are computed but not actually applied to reorder the final result set
This is a common integration bug: the reranking step runs, scores are logged or returned, but the downstream code that assembles the final context for the LLM still uses the original vector-search ordering, so the reranker appears to have no effect even though it technically executed correctly.
Fix: Add an explicit assertion or test that the final chunk order passed to the LLM matches the reranker's score-sorted order, not the original retrieval order, and add a regression test to catch this class of bug in the future.
Latency budget forces the reranker to run on too small a candidate set to be effective
Cross-encoder reranking scores each query-document pair individually rather than in the batched, precomputed way vector search does, so it scales roughly linearly with candidate count and can quickly exceed a tight latency budget. Under latency pressure, teams sometimes shrink the candidate pool so much that the reranker is back to reordering too few candidates to add value.
Fix: Profile actual reranker latency per candidate on your target hardware, and size the candidate pool to the largest value that fits your latency budget rather than an arbitrary small number; consider a faster or distilled reranker model if the budget is tight.
Diagnostic commands
Measure recall@k with and without reranking on a labeled evaluation set
python eval_retrieval.py --rerank=false && python eval_retrieval.py --rerank=true
If accuracy is statistically indistinguishable between the two runs, either the candidate pool is too small for reranking to matter, the reranker model is not well matched to the domain, or the reranker's output is not actually wired into the final result selection.
Compare the reranker's chosen top result against the original vector search top result across many queries
python compare_rankings.py --queries eval_set.jsonl
If the reranked top-1 is identical to the vector search top-1 for nearly every query, either the reranker is not changing anything meaningful, a wiring bug, or the vector search was already close to optimal for your candidate pool size.
Measure end-to-end reranking latency per query at your production candidate pool size
python -c "import time; t=time.time(); rerank(query, candidates); print(time.time()-t)"
If latency is unexpectedly high, the candidate pool may be too large for the deployed reranker model on current hardware; consider a smaller or distilled model before shrinking the candidate pool as a workaround.
Stopping it from happening again
- Always retrieve a meaningfully wider candidate pool, 20-50, than your final top-k before applying a reranker.
- Benchmark reranker model choice against a labeled, in-domain evaluation set rather than assuming a popular general-purpose model will transfer well.
- Add an automated test asserting that reranker output actually determines final chunk ordering, to catch silent wiring bugs.
- Track reranking's specific contribution to accuracy as its own metric, separate from overall pipeline accuracy, so regressions are caught early.
- Profile reranker latency against your production candidate pool size before deploying, not after users notice slow responses.
When this becomes an architecture problem
If a well-integrated reranker with an appropriately wide candidate pool still isn't improving accuracy on domain-specific queries, the bottleneck has likely shifted upstream to embedding quality or downstream to prompt grounding, and further reranker tuning has diminishing returns; that's the point to evaluate domain-adapted embeddings or a broader retrieval architecture review rather than continuing to swap reranker models.
Frequently asked questions
How many candidates should I retrieve before reranking?
20 to 50 is a reasonable starting range for most corpora; too few gives the reranker nothing meaningful to correct, while too many adds latency without much additional accuracy benefit since the correct answer is very unlikely to rank below position 50 by a reasonably good bi-encoder in the first place.
Are cross-encoder rerankers always better than bi-encoder vector search alone?
For final ranking precision at the top of the list, yes, cross-encoders that jointly process the query and document typically outperform bi-encoder similarity alone. But they are too slow to run over an entire corpus, which is exactly why the standard pattern is bi-encoder retrieval for broad recall followed by cross-encoder reranking for precision on a smaller candidate set.
Can a reranker fix bad chunking or a bad embedding model?
No. A reranker can only reorder candidates that retrieval actually surfaces; if the correct chunk never makes it into the initial candidate pool because of poor chunking or an embedding mismatch, no reranker can promote a chunk it never sees. Fix recall-limiting issues first, then add reranking to improve precision.
Is it worth the added latency to rerank on every query?
For most enterprise RAG use cases where answer accuracy matters more than shaving tens of milliseconds off response time, yes. If latency is genuinely critical, consider reranking only when the vector search's top candidates have close similarity scores, skipping it when the top result is a clear, high-confidence match.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
RAG Accuracy Readiness Assessment
Score your retrieval-augmented generation system across eight dimensions that actually predict production accuracy, from chunking strategy to groundedness verification.
Free ToolLLM Latency Budget Planner
Break total response time into time-to-first-token, generation time, and network overhead, then see your exact margin or shortfall against a target SLA.
Free ToolRAG Chunking Strategy Calculator
Turn corpus size, chunk length, and overlap into a concrete chunk count, embedding cost, and vector storage footprint before you build the ingestion pipeline.
Related problems
RAG retrieves irrelevant or wrong documents
RAG retrieves the wrong documents most often because the embedding model used to index the corpus differs from the one used at query time, or because chunks are large enough that a single embedding averages away the passage that actually answers the question. Fix embedding consistency and chunk granularity first, then add a reranker and metadata filters before touching the LLM prompt.
RAG hallucinates even though the correct context was retrieved
RAG hallucination with good context in hand usually means the prompt never explicitly instructs the model to answer only from the provided passages, or the correct passage is buried in the middle of a long context window where attention is weakest. Add an explicit grounding instruction, place the most relevant passage first, resolve conflicting retrieved chunks before generation, and give the model an explicit abstain option.
Embedding dimension mismatch after switching models
Different embedding models produce vectors of different fixed dimensions, so swapping models without re-embedding the entire corpus produces a hard dimension mismatch error or, worse, silently meaningless similarity scores if the column is resized without re-indexing. There is no shortcut: changing the embedding model requires re-embedding every document and rebuilding the vector index from scratch.
GuideHybrid Search: Combining BM25 and Vector Retrieval with RRF
Hybrid search for RAG: why pure vector retrieval misses exact matches, how BM25 fixes it, and how reciprocal rank fusion combines both reliably.
GuideEnterprise RAG Architecture: The Full 2026 Blueprint
A practitioner's blueprint for enterprise RAG in 2026: ingestion, chunking, embedding, retrieval, rerank, generation, and the eval loop that keeps it honest.
GuideRAG Evaluation Metrics: Recall@k, MRR, Faithfulness, and More
The RAG evaluation metrics that matter: recall@k and MRR for retrieval, faithfulness and answer relevance for generation, and how to build the eval loop.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.