Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
The fastest way to improve an LLM system is not to guess at better prompts — it's to read a pile of real failures, cluster them into categories, and fix the biggest cluster first. This loop, sometimes called error analysis or the data flywheel, is where practitioners spend most of their high-value time. It's unglamorous: you literally sit and read transcripts. But it's the only reliable way to discover the failure modes you didn't anticipate, and it turns a vague 'the bot is bad sometimes' into a ranked, actionable backlog. Every metric and judge you'll build later is downstream of the categories this process reveals, which is why we teach it before any of the fancy machinery.
The demo takes a batch of graded outputs, buckets the failures by an error label, and prints them ranked by frequency — so you know exactly which failure mode to attack first for the biggest score gain.
hallucinated_fact is the top cluster; note that fixing it would eliminate half the failures.wrong_tool and re-run — watch the ranking reorder.share_cumulative field so you can see how many error types you'd need to fix to cover 80% of failures (a Pareto view).hallucinated_fact case as a new category ambiguous_input and observe how splitting a bucket changes priorities.from collections import Counter
# Real failures after a run, each hand-tagged with an error category during
# error analysis. (In practice you'd tag these while reading transcripts.)
FAILURES = [
{"input": "cancel my plan", "error": "wrong_tool"},
{"input": "where is my order", "error": "hallucinated_fact"},
{"input": "upgrade to pro", "error": "wrong_tool"},
{"input": "gift card balance", "error": "hallucinated_fact"},
{"input": "refund status", "error": "hallucinated_fact"},
{"input": "change my email", "error": "ignored_instruction"},
]
def rank_errors(failures):
counts = Counter(f["error"] for f in failures)
total = sum(counts.values())
return [
{"error": err, "count": n, "share": n / total}
for err, n in counts.most_common()
]
for row in rank_errors(FAILURES):
print(f"{row['error']:>22} {row['count']} ({row['share']:.0%} of failures)")
# Fix 'hallucinated_fact' first: it's 50% of failures, one grounding fix clears it.python3 main.py