Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
There are two fundamentally different places you can measure quality, and confusing them wastes weeks. Offline evaluation runs a fixed dataset through your system before you ship — fast, cheap, reproducible, and blind to the messiness of real traffic. Online evaluation measures the system on live users after you ship — slow, noisy, expensive, but the only thing that tells you whether the feature actually works in the wild. Beginners over-index on one or the other: they either ship on offline scores that don't predict real behavior, or they fly blind in production with no gate to catch regressions before rollout. You need both, wired into a loop, and you need to know which question each one answers.
The sketch below shows the same metric — answer helpfulness — measured in both regimes. Offline, you control the inputs and know the expected answers. Online, you can't know the 'right' answer for a live query, so you fall back to proxy signals (thumbs, escalation rate, edit distance on the user's follow-up).
expected_topic the naive router gets wrong, and confirm the offline gate score falls below 1.0.dissatisfaction lands near 0.15.offline_eval can be run in CI but OnlineMonitor.report() cannot.latency_p95 proxy to the online monitor and note that offline eval has no equivalent for real-world tail latency.# OFFLINE: fixed dataset, known-good answers, deterministic score
GOLDEN = [
{"q": "reset password", "expected_topic": "account"},
{"q": "double charged", "expected_topic": "billing"},
]
def offline_eval(route_fn):
correct = sum(route_fn(c["q"]) == c["expected_topic"] for c in GOLDEN)
return correct / len(GOLDEN) # runs in CI, same number every time
# ONLINE: live traffic, no ground truth, proxy signals instead
class OnlineMonitor:
def __init__(self):
self.total = 0
self.thumbs_down = 0
self.escalated = 0
def record(self, feedback: str | None, escalated: bool):
self.total += 1
if feedback == "down":
self.thumbs_down += 1
if escalated:
self.escalated += 1
def report(self):
# proxies for quality — not ground truth, but directional
return {
"dissatisfaction": self.thumbs_down / max(self.total, 1),
"escalation_rate": self.escalated / max(self.total, 1),
}
print("offline gate:", offline_eval(lambda q: "billing" if "charge" in q else "account"))
m = OnlineMonitor(); m.record("down", False); m.record(None, True); m.record("up", False)
print("online proxies:", m.report())python3 main.py