Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
Most teams ship an LLM feature, watch it break in ways they can't reproduce, and then bolt on evaluation as damage control. That order is backwards and it costs you. Without a measurement you trust, every prompt tweak is a coin flip, every model upgrade is a gamble, and every 'it feels better now' is unfalsifiable. Adopting an eval-first mindset means you write the way you'll grade output before you obsess over the prompt — because the grader is what turns a vibe into a number you can defend to your team, your users, and yourself. This is the single highest-leverage habit change in applied LLM work, and it is why the whole rest of this course exists.
The contrast below is the entire philosophy in one frame. The 'vibes' loop changes a prompt and eyeballs one output. The eval-first loop pins a small labeled set, defines a scoring function, and reports a number every time — so a change is either an improvement or a regression, never a shrug.
CASES whose must_include list contains a keyword neither prompt produces; confirm the average score drops.score to return a strict 0/1 (all keywords required) instead of a fraction, and observe how the baseline vs improved delta changes.# The vibes loop (what most teams do)
def vibes_loop(prompt):
out = call_model(prompt, "Summarize this ticket: printer won't connect")
print(out) # you read it, nod, ship it
# No number. No record. No way to compare tomorrow's prompt to today's.
# The eval-first loop (what you'll build in this course)
CASES = [
{"input": "printer won't connect", "must_include": ["network", "driver"]},
{"input": "refund my last order", "must_include": ["order id", "refund"]},
]
def score(output, case):
hits = sum(1 for kw in case["must_include"] if kw in output.lower())
return hits / len(case["must_include"]) # 0.0 .. 1.0
def eval_first_loop(prompt):
scores = [score(call_model(prompt, c["input"]), c) for c in CASES]
return sum(scores) / len(scores) # one comparable number
# Now a prompt change is measurable:
baseline = eval_first_loop("Summarize: {input}")
improved = eval_first_loop("Summarize the support ticket, naming the subsystem: {input}")
print(f"baseline={baseline:.2f} improved={improved:.2f} delta={improved-baseline:+.2f}")python3 main.py