RAG Evaluation System
The complete harness from the RAG Evaluation Lab project
Python · Ragas · DuckDB · pgvector · OpenAI API
What it does
Runs a matrix of RAG configurations against your golden question set and reports retrieval and generation quality separately, so you can tell which half of the pipeline is costing you accuracy. Results are stored with their full configuration, and a CI mode fails a build when quality regresses past a threshold you set.
Who it is for
Teams running a RAG system in production who are currently tuning it by intuition. If you have ever changed a chunk size and shipped based on three test questions, this is the missing piece.
The principle#
The system is the product. This is not a repository of scripts — it is the complete measurement approach from the project, packaged so you can point it at your own corpus on the first afternoon.
Licence#
Single-team commercial licence. Use it in client work; do not resell it as a product.
Requirements
Python 3.11+, an embedding model endpoint, and a document corpus. PostgreSQL with pgvector optional.
Questions
How is this different from the free project?
The free project demonstrates the approach on one corpus. This is the packaged version: multi-corpus support, the CI regression gate, the annotation tool, and the judge calibration workflow.
Does it lock me into a vendor?
No. Embedding and generation providers sit behind an interface with implementations for the major APIs and for local models.
What if I do not have a golden set?
The annotation tool and a documented process for building one from real query logs are included. Expect to spend a day on your first fifty questions.
Changelog
- v2.1.014 Jul 2026
Judge calibration workflow; per-category reporting.
- v2.0.02 Apr 2026
Multi-corpus support; DuckDB results store replaces CSV.
- v1.3.020 Jan 2026
Reranking comparison; CI regression gate.
How this was built
RAG Evaluation Lab
Most RAG systems are tuned by vibes. This is a reproducible harness that measures retrieval and generation separately, so you can see which half is failing.