Turn LLM quality from a vibe into a number you can defend, gate on, and improve. Datasets, metrics, LLM judges, RAG and agent evals, red-teaming, online observability, and eval-driven development.
Ten modules that build the full evaluation and reliability stack for LLM applications, in the order a working practitioner actually builds it. You start with the eval-first mindset and the statistics of small samples, then construct trustworthy datasets and ground truth, choose and combine the metrics that matter, and build calibrated LLM-as-judge scorers. From there you assemble repeatable harnesses wired into CI release gates, then specialize into RAG and retrieval evals, agent and tool-use evals, and adversarial red-teaming. The final modules move online — tracing, production sampling, human review, drift detection, guardrails, and A/B testing — and close with eval-driven development: gating strategy, cost and latency budgets, dashboards, regression triage, and the full eval-program architecture. Python-first throughout, with every technique shown as runnable code you can lift into your own system.
Built by Lakshya Kumar
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyComplete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Build a complete eval pack and CI gate for a real (or realistic) LLM app. It must include a versioned golden set, a validated metric suite (with at least one calibrated LLM judge), cost/latency budgets, and a tiered release gate wired into CI. Prove it works: plant a genuine regression in a branch (a prompt or model change that degrades a real behavior), open a PR, and show the gate blocks the merge and the report localizes the failing cases. Submit the repo, the CI run showing the block, and a short writeup of the regression and how the gate caught it.
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
I'm taking an "AI Evaluation & Reliability Engineering" course. It covers, in order: the eval-first mindset and small-sample statistics; datasets and ground truth (golden sets, sampling, labeling, synthetic data, leakage); metrics (deterministic, semantic, precision/recall/F1, calibration, significance); LLM-as-judge (rubrics, pairwise, bias, judge-human agreement); harnesses and CI release gates (promptfoo/Braintrust patterns); RAG and retrieval evals (the triad, faithfulness, citations); agent and tool-use evals (trajectories, sandboxes); red-teaming and adversarial evals; online eval and observability (tracing, sampling, drift, guardrails, A/B); and eval-driven development (gating, budgets, dashboards, the full program). My context: 1. The LLM app I want to make reliable is: [describe — chatbot / RAG / agent / classifier] 2. My current evaluation situation: [none / I eyeball outputs / some offline tests / I have a CI gate] 3. My biggest quality problem right now: [hallucinations / inconsistency / cost / latency / safety / I don't even know] 4. My stack: [language, model provider, any eval tools already in use] Given that, answer: - Which module should I start with, and why? - Name the first three eval cases I should hand-label for my app. - Name the cheapest scorer that would catch my biggest quality problem (deterministic, semantic, or LLM-judge) and why. - If I only had one afternoon, what single eval-driven habit would give me the biggest reliability lift? - Name one thing this course will NOT solve for me, so I have the right expectations.
Build a standalone RAG evaluation suite: a golden set with deliberate hard negatives, the full triad (context relevance, faithfulness, answer correctness), retrieval metrics (recall@k, MRR), citation-accuracy checks, and an end-to-end diagnostic that localizes each failure to a pipeline stage. Run it against a real or open RAG system and submit the report plus a ranked list of the highest-leverage fixes.
Build an agent eval harness in a deterministic sandbox (task-success verifiers, tool-call correctness, trajectory scoring, operational health) plus an adversarial regression suite (injection/jailbreak, safety two-sided, robustness, fairness) fed by an automated attacker loop. Wire both as a combined gate where a safety violation is disqualifying, and submit the harness, the attacker's harvested breaches, and per-category resist rates.
Error analysis, human review, and the data flywheel. Modules 1, 2, and 9.