Diagnose a RAG system returning wrong answers

Question

Accepted Answer

Measure retrieval in isolation, first. There are only two possibilities: 1. The correct passage is never retrieved. No prompt can fix this — the information is not in the context. 2. The correct passage is retrieved and the model ignores or misreads it. Two weeks of prompt engineering only addresses the second, and the fact that it produced no improvement is itself weak evidence for the first. The diagnostic, in order: 1. Take 30 questions with known answers. Annotate which passage contains each answer. 2. Compute recall@5 — is the correct passage in the top five results? 3. If recall is low, stop and fix retrieval. Reranking is usually the highest-return change, followed by a structure-aware chunking strategy. 4. If recall is high, feed the gold passage directly and check the answer. If it is still wrong, the problem is genuinely generation. Why teams skip this: measuring retrieval requires building an annotated set, which takes a day and feels like a detour. Prompt engineering feels…

Diagnose a RAG system returning wrong answers

Go deeper on this

Why Your RAG Evaluation Is Wrong