Why Your RAG Evaluation Is Wrong

Almost every RAG evaluation I have reviewed had at least two of the following four problems. Each one makes the system look better than it is, which is why they survive: nobody goes looking for a bug that flatters them.

1. Measuring the pipeline end to end and nothing else#

A single answer-quality score cannot tell you which half of your system failed. There are two entirely different failure modes hiding behind one number:

• The passage was never retrieved. No prompt will fix this. The information is not in the context.
• The passage was retrieved and ignored. No amount of retrieval tuning will fix this.

They demand opposite work, and an aggregate score hides which one you have. Measure retrieval on its own — recall@k, MRR, hit rate, with no generation involved — before you measure anything else.

2. Evaluating on questions you wrote while building#

Questions written by the person who built the system are contaminated in a specific way: they are phrased the way the system expects. Real users ask sideways. They use internal jargon, abbreviate, and ask compound questions that span three documents.

An evaluation set drawn from your own head measures how well the system handles your phrasing. Pull questions from real logs, support tickets, or — at minimum — have someone who did not build it write them.

3. Using a language model as judge without checking the judge#

LLM-as-judge is a reasonable technique with a known failure mode: judges reward fluency and surface similarity to the source. An answer that copies the source phrasing scores well. An answer that is correct but reworded scores worse.

That bias interacts badly with the thing you are usually tuning. Prompt changes that make the model quote more verbatim will improve your judge score without improving the answer.

The fix is not to abandon LLM judges — it is to calibrate one. Label 50 answers by hand, run the judge on the same 50, and measure agreement. If agreement is poor, the judge scores are not evidence.

4. Reporting a mean over a set that is not homogeneous#

"Average faithfulness: 0.84" across a question set containing lookups, comparisons, aggregations and multi-hop reasoning is close to meaningless. Those categories have wildly different difficulty, and the mean moves whenever the mix changes.

Report by category:

Now the picture is actionable. Lookups are solved. Aggregation and multi-hop are not, and the recall column shows why: the passages are not being retrieved in the first place.

What a defensible evaluation looks like#

evaluate.pypython
def evaluate(config: RagConfig, questions: list[Question]) -> Report:
    # 1. Retrieval in isolation — cheapest and most diagnostic.
    retrieval = score_retrieval(config, questions)

    # 2. Generation given perfect retrieval — isolates the other half.
    generation = score_generation(config, questions, use_gold_context=True)

    # 3. End to end. The gap against the two above is where the loss is.
    end_to_end = score_pipeline(config, questions)

    return Report(
        retrieval=retrieval,
        generation=generation,
        end_to_end=end_to_end,
        by_category=group_by(questions, lambda q: q.category),
    )

Three numbers instead of one, reported per category. It is more work to set up once, and it is the difference between tuning a system and guessing at it.

The uncomfortable part#

Doing this properly will usually reveal that your system is worse than you thought. That is the value. An evaluation that only ever confirms your system is good is not an evaluation — it is a reassurance mechanism, and it will let a regression reach production untouched.