Better RAG Retrieval: Chunking, Hybrid Search and Reranking Tips

A company builds a chatbot on its own documents, tries ten questions, and gets mostly good answers. Then real users arrive and ask things in unexpected ways. The bot misses a clearly relevant policy, quotes an outdated page, or answers confidently from the wrong section. The language model is rarely the culprit. The retrieval step, which decides what text the model gets to read, usually is.

This guide explains how to improve retrieval quality in a retrieval-augmented generation (RAG) system. We cover chunking, hybrid search, reranking, metadata filters, and the evaluation habits that keep quality from silently slipping. It assumes you already understand the basic RAG idea and want it to work reliably.

Why retrieval is the real bottleneck

A language model can only answer from what you hand it. If the right passage is not in the retrieved context, even the best model will guess or say it does not know. If the context is full of loosely related text, the model may latch onto the wrong detail. So the quality of a RAG system is bounded by the quality of retrieval.

That is good news, because retrieval is something you can measure and tune with ordinary engineering. If you are still at the stage of building your first version, our guide to building a RAG chatbot on your company documents covers the foundations before you start tuning.

Chunking: how you cut the documents

Before search happens, documents are split into chunks. How you split them has a large effect on what can be found.

• Too small: A tiny chunk may lose the context that gives a sentence meaning, such as which product or region a rule applies to.
• Too large: A big chunk mixes topics, dilutes the match, and wastes the model's limited context window.
• Structure-aware splitting: Splitting along headings, sections, and list boundaries usually works better than cutting every fixed number of characters.
• Overlap: A small overlap between neighbouring chunks helps when an answer straddles a boundary.
• Context headers: Prepending the document title and section heading to each chunk helps retrieval and helps the model cite the source.

There is no universal chunk size. Test a few options on your own questions and documents, and let results choose, not habit.

Hybrid search: meaning plus exact words

Vector search, which compares the meaning of text using embeddings, is good at matching paraphrases. It is weaker at exact tokens such as product codes, error numbers, names, and acronyms. Keyword search, such as BM25, is the opposite: precise on exact terms, blind to paraphrase.

Hybrid search runs both and merges the results, often with a method such as reciprocal rank fusion. This combination tends to be more robust across the kinds of questions real users ask, since some are conceptual and some are literal lookups. Many search engines and vector databases now support hybrid queries directly.

Reranking: a second, closer look

First-stage retrieval is built for speed, so it returns a broad candidate set, perhaps the top twenty to fifty passages. A reranker is a model that reads the question and each candidate together and scores how well the passage actually answers it. You then keep only the best few for the language model.

Reranking adds a little latency and cost, but it can notably improve precision because it looks at the question and passage jointly rather than comparing two separate summaries. It is often one of the highest-impact upgrades once the basics are in place.

Metadata and filters

Not every question should search every document. Attaching metadata such as department, product, region, document type, version, and effective date lets you narrow the search before matching. A question from a user in one region should not retrieve a policy for another. An old version of a price list should not outrank the current one.

Filters also support access control. Make sure a user can only retrieve documents they are allowed to see, and enforce that at the retrieval layer rather than hoping the model hides restricted content.

An illustrative example

Imagine a hypothetical logistics SME whose support assistant answers questions from rate sheets, service terms, and onboarding documents. Users often ask things like "What is the surcharge for a remote pincode?" The first version uses vector search only and sometimes returns a general section on surcharges instead of the specific table. It also surfaces last year's rate sheet.

The team makes four changes. They split documents by heading and keep tables intact, add keyword search for exact terms like pincode and service names, rerank the top candidates, and tag each file with an effective date so outdated sheets are filtered out. In a scenario like this, the answers would typically become both more precise and more current. This is an illustration of the approach, not a reported client result.

A numbered tuning process

1. Collect real questions. Gather a set of genuine user questions, ideally fifty or more, covering common cases and tricky ones.
2. Mark the right sources. For each question, note which document passages contain the correct answer. This is your ground truth.
3. Measure retrieval alone. Check how often the right passage appears in the top results, before involving the language model. Track simple measures such as hit rate and rank of the first correct passage.
4. Tune chunking. Try structure-aware splits and a few sizes. Keep tables, code blocks, and lists whole.
5. Add hybrid search. Combine vector and keyword retrieval and compare against your baseline.
6. Add a reranker. Retrieve a broader set, rerank, and keep the top few. Watch latency as well as accuracy.
7. Apply metadata filters. Add version, region, and permission filters, and confirm that restricted content never leaks.
8. Evaluate end-to-end. Check answer correctness and citations, then lock the set in as a regression suite. Our guide to LLM evals and regression testing for AI features explains how to build one.

Handling freshness and updates

Documents change, and an index that is not updated becomes a liability. Set up a pipeline that re-processes a document when it changes, removes deleted files, and records the version that each chunk came from. Show the source and date in answers so users can judge freshness themselves. For fast-changing content, consider more frequent index refreshes or a hybrid approach that checks a live source for the latest values.

Key benefits of better retrieval

• More accurate answers: The model sees the right evidence and has less reason to guess.
• Fewer hallucinations: Good context plus an instruction to say "I do not know" when evidence is missing reduces invented answers.
• Lower cost: Sending a few precise passages instead of many loosely related ones uses fewer tokens.
• User trust: Citations to current, relevant sources make answers easier to verify.
• Safer access: Metadata filters help keep confidential material away from the wrong users.

Common mistakes

The most common mistake is blaming the model and swapping it for a bigger one before checking retrieval. The second is tuning by anecdote, changing settings after one bad answer without measuring the effect on the rest. The third is ignoring document quality: scanned files with poor text extraction, duplicated pages, and contradictory versions will hurt results no matter how clever the search is. Clean the corpus first.

Another pitfall is forgetting the "no answer" path. When retrieval finds nothing relevant, the assistant should say so and offer a way to reach a human, instead of producing a plausible guess.

When to bring in specialists

Many teams can handle a basic RAG prototype in-house. Production-grade retrieval, with evaluation sets, access control, monitoring, and integration with your systems, is where experienced help pays off. Our AI development service covers designing and tuning retrieval systems for business use, from document assistants to internal knowledge search.

Monitoring retrieval in production

Quality work does not end at launch. Real users will ask questions your test set never imagined, so log each query, the passages retrieved, and the final answer, with appropriate privacy controls. Review a sample regularly, especially questions that received negative feedback or ended with a request for a human. These reviews reveal gaps in the documents, missing metadata, and queries that need better handling.

Add a simple feedback control to each answer so users can flag a poor response, and treat those flags as new test cases. When a failure is fixed, add the question to your evaluation set so it cannot quietly return. Over time, this loop builds a set of examples that reflects how your users actually speak, which is more valuable than any generic benchmark.

Query understanding before retrieval

Users rarely write ideal search queries. They use short fragments, abbreviations, and follow-up questions that depend on earlier messages. A lightweight step that rewrites the question into a clear, standalone query, expands abbreviations, and resolves references to earlier turns can improve results noticeably. For complex questions, breaking them into smaller sub-questions and retrieving for each one often works better than one broad search.

Be careful, however, to keep the original wording available. Rewriting can drift from the user's intent, so compare results with and without it on your evaluation set before you adopt it. As with every other lever in this article, measure first, then keep only what demonstrably helps.

Conclusion

If your RAG assistant disappoints, look at retrieval first. Cut documents along their natural structure, combine meaning-based and keyword search, rerank the candidates, filter by metadata, and measure everything against real questions. These steps are practical, testable, and do not require a bigger model. Build a small evaluation set this week, record your baseline, and improve one lever at a time. Steady, measured tuning is what turns an impressive demo into a tool people rely on.

Frequently Asked Questions

Why does my RAG chatbot give wrong answers?

Often the retrieval step failed to surface the right passage, so the model guessed from loosely related text. Check retrieval quality before changing the language model.

What is hybrid search in RAG?

Hybrid search combines vector search, which matches meaning, with keyword search, which matches exact terms. Merging both usually handles a wider range of questions.

What does a reranker do?

A reranker reads the question and each candidate passage together and scores how well it answers the question. You then pass only the best few passages to the language model.

How big should chunks be?

There is no universal size. Split along headings and natural structure, keep tables whole, and test a few sizes against your own questions to see what retrieves best.

How do I measure retrieval quality?

Build a set of real questions with known correct passages, then track how often and how high the right passage appears in the results. Re-run it after every change.