Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
An eval you can't reproduce is an anecdote, not a measurement. LLM calls are nondeterministic by default — temperature, sampling, and even server-side changes mean the same input can yield different outputs run to run — so if you don't pin what you can pin, your eval numbers wobble and you can't tell a real regression from noise. Learning which knobs to fix (temperature, seed where supported, model version, prompt template hash, dataset version) and which sources of variance are irreducible is what makes your eval trustworthy enough to gate a release on. Teams that skip this waste days chasing 'regressions' that were just sampling noise.
The demo separates the variance you can control from the variance you can't. It pins temperature to 0 and records the exact model version and prompt hash alongside every score, so a later diff can prove whether the system or just the noise changed.
prompt_hash, correctly flagging the runs as not directly comparable.model_version and write a comment on why that silently breaks reproducibility when the alias updates.git_sha field to the fingerprint and argue in a comment why the scoring code version also matters.Working within free-tier limits. Free / low-tier provider keys rate-limit aggressively, and eval or agent loops that fan out calls will hit
429 Too Many Requestsfast. Survive it: readRetry-Afterand thex-ratelimit-*headers and back off (exponential backoff with jitter + a max-retry cap) instead of hammering; cap in-flight requests with a small concurrency limiter so you stay under the RPM/TPM ceiling; cache identical requests so retries don't re-spend quota; downshift to a smaller/cheaper model for practice runs; use the provider Batch API for non-interactive jobs; or sidestep hosted limits entirely by running a small model locally (Ollama / llama.cpp) or on a free Colab/Kaggle GPU while you learn.
import hashlib, json
# Everything that must be pinned to make a score comparable across runs.
def eval_fingerprint(model_version, prompt_template, dataset_version, temperature):
prompt_hash = hashlib.sha256(prompt_template.encode()).hexdigest()[:12]
return {
"model_version": model_version, # e.g. "claude-sonnet-4-6" (NOT an alias)
"prompt_hash": prompt_hash, # pin the exact template
"dataset_version": dataset_version,
"temperature": temperature, # 0 for deterministic-ish scoring runs
}
fp = eval_fingerprint("claude-sonnet-4-6", "Summarize: {input}", "v3", 0.0)
print(json.dumps(fp, indent=2))
# A score is only comparable to another score with the SAME fingerprint.
def comparable(fp_a, fp_b):
return fp_a == fp_b
old = eval_fingerprint("claude-sonnet-4-6", "Summarize: {input}", "v3", 0.0)
new = eval_fingerprint("claude-sonnet-4-6", "Summarize concisely: {input}", "v3", 0.0)
print("comparable?", comparable(old, new)) # False: prompt changed -> expected!python3 main.py