Why RAG Systems Fail in Production (And How to Debug Them)

The first time a RAG system I built gave a confidently wrong answer in front of a client, my instinct was to blame the model. I swapped it for a bigger one. The answer got more confident and more wrong. The model was never the problem. The retriever was handing it the wrong chunks, and no amount of model quality fixes that.

This is the pattern I see most often when people debug RAG systems: they tune the prompt, they swap the model, they add more context. What they almost never do first is check whether the retriever is actually finding the right documents. So that's where this post starts.

Start By Separating Retrieval From Generation#

The single most useful debugging step is also the one people skip: pull the retrieved chunks out and read them yourself, before the LLM ever sees them.

def debug_retrieval(query: str, retriever, k: int = 5):
    """Print retrieved chunks with their scores, no LLM involved."""
    results = retriever.similarity_search_with_score(query, k=k)
    for i, (doc, score) in enumerate(results):
        print(f"\n--- Result {i+1} (score: {score:.4f}) ---")
        print(f"Source: {doc.metadata.get('source', 'unknown')}")
        print(doc.page_content[:300])
    return results

Run this on ten real user queries from your logs, not synthetic test cases. If the right chunk isn't in the top five, you've found your bug and you haven't touched the LLM yet.

The Chunking Problem Hiding in Plain Sight#

A huge share of "the retriever is bad" complaints are actually "the chunks are bad" complaints. Fixed-size chunking (split every 500 characters, whatever falls where it falls) routinely cuts a sentence in half, separates a table header from its rows, or splits a code block mid-function. The embedding for that broken chunk is now a blurry average of two unrelated ideas, and no similarity search will find it reliably.

from langchain.text_splitter import RecursiveCharacterTextSplitter
 
# Better: split on structure first, size second
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""],  # try paragraph breaks first
)
 
chunks = splitter.split_documents(documents)

The separators list matters more than the chunk_size number most people obsess over. Splitting on paragraph and sentence boundaries first, and only falling back to a hard character cutoff as a last resort, fixes a surprising number of "irrelevant chunks" complaints on its own.

Check Whether You're Even Measuring The Right Thing#

It's easy to eyeball a few results, decide they look "close enough," and move on. A better habit is to build a small labeled evaluation set: twenty to fifty real queries with the document ID you know is the correct answer, and measure recall directly.

def evaluate_recall_at_k(eval_set, retriever, k: int = 5):
    """eval_set: list of (query, expected_doc_id) pairs."""
    hits = 0
    for query, expected_id in eval_set:
        results = retriever.similarity_search(query, k=k)
        retrieved_ids = [doc.metadata.get("id") for doc in results]
        if expected_id in retrieved_ids:
            hits += 1
    recall = hits / len(eval_set)
    print(f"Recall@{k}: {recall:.2%} ({hits}/{len(eval_set)})")
    return recall

This is the single highest-leverage thing you can build for a RAG system, because it turns "I think retrieval got better after that change" into a number you can actually compare across changes. Twenty labeled examples is enough to start noticing trends.

Hybrid Search Fixes A Specific, Common Failure Mode#

Pure vector similarity search struggles with exact terms: product codes, error messages, acronyms, proper nouns. If a user searches for "error E4021" and your documents contain that exact string, a dense embedding model may not weight the literal match highly enough, because semantically "E4021" doesn't mean much to it.

from rank_bm25 import BM25Okapi
 
class HybridRetriever:
    def __init__(self, documents, vector_store, alpha: float = 0.5):
        self.vector_store = vector_store
        self.documents = documents
        tokenized = [doc.page_content.lower().split() for doc in documents]
        self.bm25 = BM25Okapi(tokenized)
        self.alpha = alpha  # weight toward vector (1.0) vs keyword (0.0) search
 
    def search(self, query: str, k: int = 5):
        vector_results = self.vector_store.similarity_search_with_score(query, k=k * 2)
        bm25_scores = self.bm25.get_scores(query.lower().split())
 
        # Normalize both score sets to 0-1, then blend
        combined = {}
        for doc, score in vector_results:
            doc_id = doc.metadata.get("id")
            combined[doc_id] = self.alpha * (1 - score)  # lower distance = better
 
        for i, score in enumerate(bm25_scores):
            doc_id = self.documents[i].metadata.get("id")
            existing = combined.get(doc_id, 0)
            combined[doc_id] = existing + (1 - self.alpha) * score
 
        ranked = sorted(combined.items(), key=lambda x: x[1], reverse=True)
        return ranked[:k]

Blending BM25 keyword scoring with vector search catches exact-match cases the embedding model glosses over, while still keeping the semantic matching that makes vector search useful in the first place. Starting with alpha=0.5 and adjusting based on your recall evaluation set is a reasonable default.

Reranking: The Step Most Pipelines Skip#

Even a good hybrid retriever pulls back k=10 or k=20 candidates that are only roughly ranked. A cross-encoder reranker looks at the query and each candidate document together, rather than comparing independent embeddings, and produces a much sharper final ordering.

from sentence_transformers import CrossEncoder
 
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
 
def rerank(query: str, candidates: list, top_n: int = 5):
    pairs = [[query, doc.page_content] for doc in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [doc for doc, score in ranked[:top_n]]

This adds latency (a cross-encoder pass over 20 candidates typically costs 50 to 150ms on a decent GPU, more on CPU), so it's worth measuring whether the accuracy gain is worth it for your use case. For anything where a wrong answer has a real cost, it usually is.

A Debugging Checklist I Actually Use#

When a RAG system misbehaves in production, I go through these in order, because each one rules out a whole category of bugs before moving to the next:

1. Read the raw retrieved chunks for the failing query. Is the right information even in there?
2. Check the chunk boundaries. Did chunking cut the answer in half or separate it from its context?
3. Try the query as a literal keyword search. If BM25 alone finds it but vector search doesn't, you likely need hybrid search.
4. Measure recall@k on a labeled set, not just this one query. One bad result might be noise; a low recall score is a real problem.
5. Only then look at the prompt and the model. If retrieval is solid and the answer is still wrong, that's where the actual generation debugging starts.

The Honest Limitations#

Hybrid search and reranking help a lot, but they don't fix everything. Multi-hop questions (where the answer requires connecting two separate documents) still need either query decomposition or a fundamentally different retrieval strategy. Very large corpora need approximate nearest neighbor indexes (HNSW, IVF) that trade a small amount of recall for a large amount of speed, and that tradeoff needs to be tuned deliberately, not left on defaults. And no retrieval fix helps if the source documents themselves are outdated or wrong, which is worth checking before you assume the pipeline is broken.

Start with the eval set. Everything else in this post is easier to justify, and easier to tune, once you can actually measure whether a change helped.