How do you improve RAG retrieval with hybrid search and reranking?

Conclusion

Vector search alone often misses:

• exact terms (IDs, error codes, product names)
• rare tokens (part numbers)
• short queries

Hybrid retrieval (keyword + vector) plus reranking is a practical upgrade. The minimum approach:

1. apply strict metadata filters first
2. run hybrid retrieval (BM25 + vectors)
3. rerank top candidates with a cross-encoder (or LLM reranker)
4. evaluate hit rate on a small test set

This reduces “wrong chunk wins” without rewriting your whole stack.

Explanation

RAG retrieval has three stages:

• candidate generation (retrieve top-N)
• reranking (choose best K)
• context assembly (what the model sees)

Vectors are good for semantic similarity. Keywords are good for exact matches. Rerankers are good at choosing the best chunk when top-N contains both good and bad candidates.

The highest leverage pattern is:

• filters → hybrid retrieve → rerank → assemble context

Practical Guide

Step 1: enforce metadata filters before retrieval (5 minutes)

Always filter by:

• tenant/org
• access labels
• product area (if available)

If filters are weak, hybrid search can amplify noise.

Step 2: configure hybrid retrieval (10 minutes)

Start with:

• BM25 top 50
• vector top 50
• union results and dedupe

Then rerank to top 5–15.

Tip:

• hybrid is especially helpful for short queries.

Step 3: add a reranker (15 minutes)

Options:

• cross-encoder reranker (fast, strong)
• LLM reranker (flexible, slower)

Reranker input should include:

• query
• chunk text
• chunk metadata (title, section)

Rule:

• reranker runs on a bounded candidate set (top-N), not the whole index

Step 4: use query rewriting (optional) (10 minutes)

Query rewriting helps when users ask vague questions.

Patterns:

• expand acronyms
• add product context
• convert to keyword-friendly terms

Guardrail:

• log rewritten query and allow disabling per route

Step 5: tune top-k and context limits (10 minutes)

Common defaults:

• retrieve N=100 candidates
• rerank to K=10
• send 3–8 chunks to the LLM

Rules:

• too many chunks = dilution
• too few chunks = missing evidence

Step 6: evaluate retrieval quality (15 minutes)

Measure:

• top-k hit rate (does the right doc appear?)
• MRR / rank position of the correct chunk
• duplication rate in top-K

Log:

• filters applied
• keyword results vs vector results
• reranker scores

Without measurement, you’re just guessing.

Pitfalls

• enabling hybrid without filters (noise explosion)
• reranking too many candidates (cost/latency)
• sending top-K chunks directly to LLM without dedupe
• rewriting queries silently without logging
• optimizing for “semantic similarity” instead of hit rate

Checklist

• [ ] Metadata filters are enforced (tenant/access/product)
• [ ] Hybrid retrieval runs BM25 + vectors and dedupes
• [ ] Candidate set size (top-N) is bounded
• [ ] A reranker selects the best K candidates
• [ ] Context assembly dedupes and limits chunks
• [ ] Query rewriting is logged and configurable
• [ ] Retrieval evaluation uses a small test set
• [ ] Metrics include hit rate and rank position (MRR)
• [ ] Logs capture keyword/vector/reranker contributions

FAQ

1) Do I need hybrid search if vectors are good?

Often yes. Exact identifiers and short queries benefit a lot from keyword retrieval.

2) What’s the fastest upgrade?

Add a reranker on top of your existing retrieval. It improves quality without changing indexing.

3) Should I use an LLM as a reranker?

If you can tolerate latency/cost, it works. For most production systems, a small cross-encoder reranker is faster and cheaper.

Internal links

• Hub: AI development
• Related:
  • RAG chunking and indexing checklist
  • RAG citations and grounding checklist
  • Minimal LLM logging checklist

Disclaimer

General engineering guidance only.