How would you evaluate a RAG system?

Three measurements, not one.

1. Retrieval alone. With an annotated set of questions and their answering passages, compute recall@k, MRR and hit rate. No generation involved — this is cheap and the most diagnostic signal in the system.

2. Generation given perfect retrieval. Feed the gold passage directly and score faithfulness and answer relevance. This isolates the generation half completely.

3. End to end. The gap between this and the two isolated measurements localises the loss precisely.

Report by question category — lookup, comparison, aggregation, multi-hop — because a mean across categories of very different difficulty moves whenever the mix changes and tells you nothing actionable.

Calibrate the judge. If you use a model as judge, hand-label 50 answers and measure agreement. Judges reward fluency and verbatim overlap with the source, which biases exactly the changes you are likely to be testing.

The strongest signal in an answer here is the separation of retrieval from generation. A candidate who proposes a single end-to-end quality score has not run a RAG system long enough to have been stuck.

Good follow-up material: how you build the golden set from real query logs, how you gate a deployment on regression, and why chunk size gets far more attention than it deserves relative to reranking.