Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
It's tempting to talk about a model's quality as one number, but post-training work forces you to split it into two axes that fail independently: capability (can the model solve the problem at all, given unlimited attempts) and alignment (does the model actually try to give you the answer you want, in the format you want, without refusing or going off the rails). A model can ace a benchmark in the few-shot setting — proving the capability is in there — and still fail the same task zero-shot in a chat interface because it wasn't post-trained to present that capability the way a user expects; that's an alignment gap, not a capability gap, and no amount of RL will fix it if the underlying pretraining never learned the skill. Conversely, a model can be extremely well-aligned (polite, on-format, honest about uncertainty) and still be wrong, because alignment training has no mechanism to teach new facts or reasoning ability it didn't already have. Diagnosing which axis broke before you touch post-training saves enormous wasted effort: alignment problems are cheap to fix with more or better SFT/RL data, capability problems usually require going back to pretraining or scaling up.
This tiny diagnostic compares a model's zero-shot answer to its best answer across several sampled attempts (a rough capability probe) against how often its default, single-shot answer matches that best answer (a rough alignment probe).
import random
def sample_answers(question, n=8):
"""Stand-in for calling a model n times with sampling temperature > 0."""
# Pretend the model 'knows' the right answer sometimes but doesn't
# reliably surface it as its top, zero-shot response.
correct = "42"
pool = [correct] * 3 + ["forty", "I'm not sure", "41", "42.0"]
random.seed(question)
return [random.choice(pool) for _ in range(n)]
def capability_probe(question, correct):
samples = sample_answers(question, n=8)
return any(s.strip() == correct for s in samples) # can it EVER produce it
def alignment_probe(question, correct):
zero_shot = sample_answers(question, n=1)[0] # its one default answer
return zero_shot.strip() == correct
q = "seed-question-1"
cap = capability_probe(q, "42")
align = alignment_probe(q, "42")
print(f"capability(ever correct)={cap} alignment(default correct)={align}")
if cap and not align:
print("-> capability gap is closed; this is an alignment problem, fixable with SFT/RL.")
elif not cap:
print("-> the model never produces the right answer; that's a capability ceiling.")python3 main.pypool so the correct answer never appears; rerun and confirm capability_probe now returns False, correctly flagging a capability ceiling instead of an alignment gap.n in capability_probe and observe how a low-probability-but-present correct answer becomes far more likely to surface — this is why capability is often measured with pass@k, not pass@1.format_probe that checks whether the answer is a bare number vs a full sentence, and reason about which axis (capability or alignment) a formatting failure belongs to.