The Instruction-Following Gap and the Alignment Tax

Why this matters

The 'instruction-following gap' is the distance between a base model's latent capability and what it actually produces when asked directly, and closing it is the headline goal of post-training. But that closing act isn't free: labs consistently observe an 'alignment tax' — small regressions on raw capability benchmarks (perplexity on held-out text, sometimes few-shot benchmark scores) after heavy RLHF, because the policy has been pulled away from the pretraining distribution toward a narrower, more constrained region of output space that satisfies human raters. This tradeoff matters practically: a small alignment tax is usually a great trade (you gain a model people can actually use), but an unmonitored one can silently erode capability you paid enormous compute to build in pretraining. The engineering discipline this produces is to always measure both sides — instruction-following quality AND capability benchmarks — before and after every post-training change, treating a capability drop as a real cost to be justified, not an invisible side effect.

Demo

The snippet below tracks both numbers side by side across a toy sequence of post-training steps, so an alignment gain that quietly costs capability shows up immediately instead of hiding in a single averaged metric.

STEPS = [
    {"step": "base",        "instruction_following": 0.20, "mmlu_like": 0.62},
    {"step": "+SFT",        "instruction_following": 0.68, "mmlu_like": 0.61},
    {"step": "+RLHF (mild)","instruction_following": 0.83, "mmlu_like": 0.60},
    {"step": "+RLHF (heavy)","instruction_following": 0.91, "mmlu_like": 0.54},
]

def report(steps):
    prev = None
    for s in steps:
        line = f"{s['step']:16s} IF={s['instruction_following']:.2f}  cap={s['mmlu_like']:.2f}"
        if prev:
            d_if  = s["instruction_following"] - prev["instruction_following"]
            d_cap = s["mmlu_like"] - prev["mmlu_like"]
            line += f"   (delta IF={d_if:+.2f}, delta cap={d_cap:+.2f})"
            if d_cap < -0.03:
                line += "  <-- alignment tax getting expensive"
        print(line)
        prev = s

report(STEPS)

Try it yourself

1. Add a fifth step where instruction-following barely improves but capability drops another 5 points, and confirm the alert fires — this is a regression you should reject.
2. Change the alert threshold and argue, in one sentence, what threshold you'd actually use in a real release-gate and why.
3. Compute the ratio of instruction-following gained to capability lost between the mild and heavy RLHF steps, and decide which one looks like the better trade.
4. Find one public model card or paper that reports a capability benchmark before and after RLHF, and note the actual delta.

References

• Ouyang et al. — Training language models to follow human feedback (InstructGPT)
• Casper et al. — Open problems and fundamental limitations of RLHF