RAG that survives production: hybrid retrieval, evals, and when GraphRAG is overkill

A RAG demo can look excellent while its production version quietly answers the wrong question. The index is stale, a query contains an exact product code that semantic search misses, the top chunk is relevant but incomplete, and nobody notices until a customer makes a decision from the answer.

Production RAG is not primarily a prompt-writing exercise. It is a retrieval and evaluation system with an LLM at the end. The durable design is usually less exotic than the architecture diagram suggests: combine complementary retrieval methods, make evidence visible, measure the failure modes you care about, and keep the simplest system that meets the requirement.

This is a companion to the practical Qdrant hybrid-search guide and the earlier RAG evaluation guide. It focuses on the decisions that appear after the first successful prototype.

Start with a failure contract

Before choosing a vector database or an orchestration framework, write down what “wrong” means for your application. A support assistant might fail by citing an obsolete policy, omitting an eligibility condition, or confidently answering outside the knowledge base. An internal engineering assistant might fail by retrieving a similar-looking but incompatible API version.

Turn those failures into explicit requirements:

• Which sources are authoritative when documents disagree?
• How fresh must an answer be?
• Must every material claim have a citation?
• What should the system do when evidence is weak or absent?
• Which users or tenants may retrieve each document?
• What latency and cost budget applies to a normal request?

These questions determine the retrieval design. They also give you a testable boundary. “The answer sounds useful” is not a production criterion; “the answer cites the current refund policy and refuses when no policy supports the claim” is.

Why hybrid retrieval is the sensible baseline

Dense retrieval represents a query and documents as vectors, then finds semantically similar passages. It is good at paraphrases: “How long may I work from another country?” can match a passage headed “temporary remote-work location.” It can also struggle with identifiers, uncommon names, error codes, version numbers, and short queries where a single token carries most of the meaning.

Sparse retrieval, commonly based on an inverted index such as BM25, scores term overlap. It is strong when the user types an exact SKU, CVE, function name, ticket number, or quoted phrase. It is less forgiving of synonyms and changes in wording.

Hybrid retrieval runs both and combines their candidates. That is not redundant work. The two methods fail differently, which is precisely what you want from a first-stage candidate generator. The Qdrant example linked above demonstrates the same principle with dense and sparse representations; the storage choice can differ, but the retrieval reasoning remains portable.

A useful request path looks like this:

1. Normalize the query without destroying meaningful identifiers.
2. Apply authorization and tenant filters before returning candidates.
3. Run sparse and dense retrieval independently, usually for more candidates than you finally display.
4. Fuse the ranked lists with a method such as reciprocal rank fusion, or use a calibrated weighted combination.
5. Rerank the fused candidates with a more expensive model or scoring step.
6. Trim redundant chunks, assemble context, and answer only from the selected evidence.

Do not assume the same top-k is correct for every stage. A first-stage search might retrieve 30 candidates from each method. Fusion can reduce that to a manageable pool, and a reranker can select the final five to eight passages. The exact numbers are workload-specific; measure them instead of copying a fashionable configuration.

Reranking earns its latency when the corpus is difficult

Initial retrieval is optimized for recall and throughput. It needs to find plausible evidence quickly. A reranker can inspect the query and candidate text together, making a more nuanced relevance decision than a single vector similarity score.

Use it where the extra latency buys something visible: long documents, near-duplicate policies, technical terminology, or questions that require a passage to satisfy several constraints at once. Do not add it simply because every architecture diagram has a reranker box. On a small, clean corpus with short documents, better chunking and metadata filters may produce a larger gain.

Measure reranking as a trade-off. Record retrieval latency, reranking latency, token volume, cache hit rate, and the change in useful-answer rate. A reranker that improves a benchmark by one percentage point while doubling p95 latency may be the wrong choice for an interactive assistant. A reranker that prevents a costly compliance error may be worth much more than its per-request price.

Chunking is part of retrieval quality

Hybrid search cannot recover information that was split badly. A chunk that contains a definition without its exception is technically relevant but operationally unsafe. A chunk that starts in the middle of a procedure may match well while leaving the user without the prerequisite or completion step.

Prefer boundaries that preserve meaning: headings, list groups, code blocks, table rows, and complete policy clauses. Store document title, section path, source, version, publication date, and access-control metadata alongside the text. Small overlap can help with transitions, but overlap is not a substitute for structure-aware splitting.

Then inspect actual misses. For each failed question, ask whether the problem was ingestion, authorization, chunk boundaries, query formulation, candidate generation, reranking, context assembly, or generation. Fix the earliest failing stage. Increasing the model size at the end often hides rather than solves a retrieval defect.

Evaluate retrieval separately from answers

A generated answer can look plausible even when the retriever missed the best evidence. Conversely, a retriever can return the right evidence while the answer model misreads it. Keep these evaluations separate so a score does not blur the diagnosis.

Build a small, versioned evaluation set from real questions and known incidents. Each case should include the expected source or acceptable sources, relevant passage boundaries where practical, and the behavior expected when evidence is absent. Include hard negatives: documents that mention the same product or topic but do not answer the question.

For retrieval, useful offline measures include:

• Recall@k: whether an acceptable passage appears in the top k.
• Precision@k: how much of the returned set is useful.
• MRR or nDCG: whether the best evidence appears near the top and how graded relevance is ordered.
• Filter correctness: whether forbidden or wrong-tenant documents ever enter the candidate set.
• Freshness correctness: whether a superseded document is outranked or excluded.

These metrics are not interchangeable. A support system may prioritize recall at the candidate stage, then use reranking and context limits to improve precision. A legal or policy workflow may require strict source and version constraints even if that reduces recall.

For answers, measure groundedness, completeness, citation correctness, refusal behavior, and instruction-following. A judge model can help triage large test sets, but it is not an oracle. Give it a rubric, the question, the retrieved evidence, and the answer; do not let it award credit for facts that are absent from the evidence. Sample and manually review judge decisions, especially for high-risk cases.

Make regressions difficult to hide

A regression suite should run when you change chunking, embeddings, metadata filters, reranking, prompts, models, or source data. Store the retrieval configuration and corpus snapshot with each run. Otherwise a score change may be caused by a moving index rather than your code.

Use three layers of checks:

1. Deterministic checks for access control, required citations, empty-result handling, schema validity, and known source/version rules.
2. Retrieval metrics against labeled questions, with thresholds and a review path for expected trade-offs.
3. Judge-assisted and human-reviewed answer checks for groundedness, completeness, tone, and refusal quality.

Keep a small “never break” set next to a broader sampled set. The small set should contain incidents that previously reached users, exact identifiers, adversarially ambiguous questions, and questions that must produce “I don’t have enough evidence.” Averages can improve while one critical behavior regresses; per-case assertions catch that.

Production telemetry should connect the trace from query to retrieved document IDs, scores, filters, reranker output, prompt version, answer, citations, latency, and cost. Redact sensitive text and enforce access controls on traces. Review low-confidence or user-corrected answers as new evaluation cases. This closes the loop without pretending that offline tests cover every future question.

GraphRAG: when it earns the complexity

GraphRAG can help when the answer depends on explicit relationships rather than similarity alone: multi-hop questions across entities, organizational or asset dependencies, changing ownership structures, or questions such as “which services depend on a component owned by the team that maintains this system?” A graph can make those relationships queryable and can support community or entity-level summaries over a large corpus.

That benefit comes with a real operating bill. You need an entity and relationship extraction strategy, identifiers that remain stable, a graph update pipeline, conflict resolution, provenance, permissions, graph-aware retrieval, and tests for extraction errors. The graph is not automatically true because an LLM extracted it. A wrong edge can be more misleading than a missed paragraph because it creates a confident path between unrelated facts.

GraphRAG is probably overkill when most questions are single-hop lookups, documents already contain the needed answer, metadata filters solve the access problem, or the corpus and team are still changing rapidly. It is also a poor first move when you have no labeled questions, no source freshness process, or no way to review extracted relationships. A graph will not compensate for weak ingestion and absent evaluation.

Use concrete signals before adopting it:

• A measured set of important questions repeatedly requires two or more retrieval hops.
• Hybrid retrieval returns the right documents, but the application cannot reliably connect their entities or constraints.
• Relationship traversal is a product requirement, not an architecture preference.
• The corpus has stable identifiers, owners, provenance, and a sustainable update path.
• You can afford to evaluate both extracted edges and final answers.

Start with a narrow graph slice or a relational adjacency table for one workflow. Compare it with a carefully tuned hybrid baseline on the same questions, latency budget, and failure review. If the graph does not improve a decision that matters, remove it. “GraphRAG” is not a maturity badge.

A production path that stays boring

A sensible rollout is incremental: establish source ownership and access filters; improve chunking and metadata; add sparse retrieval beside dense retrieval; fuse and rerank; create labeled retrieval and answer tests; then instrument the live path. Only after the baseline has measurable failures should you consider graph traversal or other specialized layers.

The best production RAG system is not the one with the most components. It is the one that can explain why a passage was selected, detect when quality changed, refuse when evidence is missing, and give its operators a practical fix. Hybrid retrieval improves coverage. Evaluation turns quality into an engineering signal. GraphRAG has a place—but only when the relationships justify the machinery.