Project · Baseline eval harness for one feature

Take one real or realistic LLM feature you care about, hand-label a 20-case golden set, and build the four-part eval (dataset, system, scorer, aggregate) that produces one comparable score. Run it, do a first pass of error analysis on the failures, and report the top failure cluster plus the Wilson confidence interval on your pass rate.

• A 20-case labeled dataset for one feature, stored as JSON or CSV with inputs and expected outputs.
• A runnable four-part harness that prints a single mean score plus per-case scores.
• An error-analysis table ranking failure categories by frequency.
• A one-paragraph writeup stating the pass rate with its 95% Wilson confidence interval and the top failure cluster to fix next.

• The harness cleanly separates dataset, system, scorer, and aggregation into distinct functions.
• The reported pass rate is accompanied by a correctly computed confidence interval, not a bare percentage.
• Failures are tagged into at least three categories and ranked by frequency.
• The writeup names one specific, actionable fix for the dominant failure cluster.