Why 'it works on my three prompts' fails

Why this matters

The most seductive trap in LLM development is the hand-picked demo. You try three prompts, they all work, you ship — and then real users send the fourth, fifth, and thousandth prompt that you never imagined. Three examples cannot estimate a rate: a feature that passes 3/3 of your favorites can easily be a 60%-reliable feature in production, and you have no way to tell the difference from that sample. Understanding the statistics of small samples — why n=3 tells you almost nothing about the true pass rate, and how wide the confidence interval is — is what stops you from shipping confidently broken things. This is the quantitative backbone of everything that follows.

Demo

The demo computes a Wilson confidence interval for an observed pass rate. Watch how catastrophically wide the interval is at n=3 versus n=100 — the same observed pass rate means wildly different things depending on how many cases you actually ran.

Try it yourself

1. Run it and read the n=3 interval aloud — note the lower bound is well under 50%, meaning 3/3 is compatible with a coin-flip feature.
2. Change the call to wilson_interval(9, 10) (9 passes of 10) and compare its interval width to wilson_interval(90, 100).
3. Find, by trial, the smallest n at which a 100% observed rate guarantees a true lower bound above 90%.
4. Add a print of the interval width (hi minus lo) for each n and confirm it shrinks roughly with 1/sqrt(n).

References

• Wilson score interval (Wikipedia)
• Evidently AI — How many samples for LLM evals

Chat about this lesson