Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
Not every error is equal, and treating them as equal is how teams optimize the wrong metric. An LLM that confidently gives wrong medical dosing information is catastrophic; the same model getting a movie recommendation slightly off is a shrug. Before you can pick metrics or set thresholds, you have to map your failure modes to their real-world cost — because a 2% error rate might be perfectly acceptable for one feature and a lawsuit for another. This framing forces you to define, in your own domain, what counts as a false positive, a false negative, and a harmful hallucination, and to weight them accordingly. Skipping this step means you'll spend effort driving a number that doesn't correspond to anything you actually care about.
The demo assigns an explicit cost to each error class and computes an expected-cost-per-query, so that two systems with the same raw accuracy can be correctly ranked by the damage they actually do.
harmful_advice cost to 5000 and note how heavily a single harmful case now dominates the ranking.missed_refund cost to 30 (angry churned customer) and re-check which system now wins.# Two classifiers with identical accuracy but very different real-world cost.
# Domain: a support bot deciding whether to auto-refund.
COSTS = {
"correct": 0, # right call, no cost
"false_refund": 25, # refunded when we shouldn't have -> lost money
"missed_refund": 5, # should have refunded, didn't -> mild annoyance
"harmful_advice": 500, # told user something dangerous -> severe
}
def expected_cost(confusion: dict) -> float:
total = sum(confusion.values())
return sum(COSTS[k] * v for k, v in confusion.items()) / total
# Both are 90% "accurate" (90 correct of 100) but distribute errors differently
cautious = {"correct": 90, "missed_refund": 9, "false_refund": 1, "harmful_advice": 0}
reckless = {"correct": 90, "missed_refund": 1, "false_refund": 8, "harmful_advice": 1}
print("cautious expected cost/query:", expected_cost(cautious))
print("reckless expected cost/query:", expected_cost(reckless))
# Same accuracy, wildly different cost. Accuracy alone would call them a tie.python3 main.py