Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
Every evaluation, no matter how fancy, decomposes into the same four parts: a dataset of inputs, a system-under-test that produces outputs, a scorer that grades those outputs, and an aggregation that rolls per-case scores into a summary. If you can name these four parts for any eval you encounter, you can debug it, extend it, and swap in your own pieces. Beginners tend to fuse these together into one tangled script, which makes it impossible to answer basic questions like 'is this the dataset's fault or the scorer's fault?' Learning to keep the four parts separate is the structural discipline that lets an eval grow from ten cases in a notebook to ten thousand in CI without a rewrite.
The code below is a minimal but complete eval with all four parts labeled and separated. This exact skeleton — dataset, system, scorer, aggregate — is the shape of every harness you'll build for the rest of the course.
system (Saturn should be Jupiter) and confirm the mean rises to 1.0 with an empty failures list.scorer with a substring-match version and observe which previously-failing cases now pass.expected the system can't produce and confirm it shows up in the failures list.run_eval to also return the per-case scores so you can see the distribution, not just the mean.from dataclasses import dataclass
# 1. DATASET — inputs plus whatever ground truth the scorer needs
@dataclass
class Case:
input: str
expected: str
DATASET = [
Case("2+2", "4"),
Case("capital of France", "Paris"),
Case("largest planet", "Jupiter"),
]
# 2. SYSTEM UNDER TEST — swap this out to compare prompts/models
def system(inp: str) -> str:
fake = {"2+2": "4", "capital of France": "Paris", "largest planet": "Saturn"}
return fake.get(inp, "unknown")
# 3. SCORER — grades one output against one case, returns 0..1
def scorer(output: str, case: Case) -> float:
return 1.0 if output.strip().lower() == case.expected.lower() else 0.0
# 4. AGGREGATE — rolls per-case scores into a report
def run_eval(dataset, system, scorer):
rows = [(c, system(c.input), ) for c in dataset]
scored = [(c, out, scorer(out, c)) for c, out in rows]
mean = sum(s for _, _, s in scored) / len(scored)
fails = [(c.input, out) for c, out, s in scored if s < 1.0]
return {"mean": mean, "n": len(scored), "failures": fails}
print(run_eval(DATASET, system, scorer))python3 main.py