Production AI Engineering: RAG, Agents, Evals & MCP
The engineering around the model
Python · LangGraph · Ragas · pgvector · MCP · Claude API · FastAPI
The gap this fills#
There is a great deal of material on prompting and very little on the engineering that makes a language-model system survive production: evaluation gates, budget control, replayable traces, and failure modes that are safe rather than confident.
Prerequisites#
Strong Python. You should have built at least one RAG prototype and been dissatisfied with it.
Curriculum
5 sections · 16 lessons
Retrieval
- Chunking strategies compared
- Reranking
- Hybrid search
- Measuring retrieval alone
Evaluation
- Building a golden set
- LLM-as-judge and its biases
- Regression gates in CI
Agents
- State machines over loops
- Budget caps and retry limits
- Failing safely
MCP
- Why a tool protocol
- Building a server
- Security boundaries
Operating it
- Cost tiering
- Tracing and replay
- Rolling out a prompt change
Free material on this topic
Why Your RAG Evaluation Is Wrong
Most RAG evaluations measure the wrong thing, on the wrong data, with a judge that rewards the wrong behaviour. Here is what to measure instead.
RAG Evaluation Lab
Most RAG systems are tuned by vibes. This is a reproducible harness that measures retrieval and generation separately, so you can see which half is failing.
AI Research & Fact-Checking System
A research agent that decomposes a claim, gathers evidence from multiple sources, cross-checks agreement between them, and refuses to answer when the evidence is thin.
AI Token Cost Calculator
Model input and output tokens, request volume and caching to project monthly cost across model tiers.