RAG Evaluation Lab

Almost every RAG system in production was tuned the same way: someone changed the chunk size, asked it three questions they remembered, decided it felt better, and shipped.

The reason is not laziness. It is that a RAG pipeline has at least six independent knobs — chunk size, overlap, embedding model, retrieval depth, reranking, prompt — and no obvious way to attribute a change in output quality to any one of them. So people stop attributing and start guessing.

A harness that treats a RAG configuration as an experiment. You define configurations declaratively, run them all against a fixed question set, and get a comparison table with retrieval metrics and generation metrics reported separately.

That separation is the point. A bad answer has two very different causes — the right passage was never retrieved, or it was retrieved and the model ignored it — and they demand opposite fixes. A single quality score hides which one you have.

Retrieval first, always#

Run the retrieval metrics before touching anything else. In every corpus tested here, retrieval recall explained more of the variance in final answer quality than every generation-side change combined.

If recall@5 is 0.62, no prompt engineering will save you. The answer is not in the context. Prompt work at that point is rearranging furniture in a room with no floor.

evaluate.pypython
def retrieval_metrics(results: list[Result], k: int = 5) -> dict[str, float]:
    hits = [r for r in results if r.gold_chunk_id in r.retrieved_ids[:k]]

    reciprocal_ranks = []
    for r in results:
        if r.gold_chunk_id in r.retrieved_ids[:k]:
            reciprocal_ranks.append(1 / (r.retrieved_ids.index(r.gold_chunk_id) + 1))
        else:
            reciprocal_ranks.append(0.0)

    return {
        "hit_rate": len(hits) / len(results),
        "mrr": sum(reciprocal_ranks) / len(results),
    }

What the sweep actually showed#

Across three corpora — technical documentation, a policy handbook, and a set of transcripts — the ranking of what mattered was consistent and not what the internet says:

Chunk size is the most-discussed parameter and, within a sensible range, close to the least important. Reranking is the least-discussed and consistently the largest single gain.

Why DuckDB#

Results live in a single DuckDB file committed alongside the corpus. No service to run, no schema migration to manage, and analytical queries across hundreds of runs return instantly. For an experiment log that one person queries interactively, anything more is overhead.