Why RLHF Exists: What SFT Alone Can't Fix

Why this matters

The naive plan — just collect more and better (prompt, ideal response) pairs and keep doing SFT — runs into a wall that isn't about data quantity. SFT is imitation learning: the model learns to reproduce the distribution of your labeled examples, which means its ceiling is bounded by how good your demonstrators are at writing responses, and it has no mechanism to express a preference between two responses neither of which any human sat down and wrote by hand. RLHF exists because ranking two candidate responses ('this one is more helpful, that one is safer') is a task humans do far more reliably and cheaply than authoring a perfect response from scratch, and because a reward model trained on those rankings can generalize the preference signal across the model's entire output distribution, not just the exact prompts you labeled. This is also why RLHF can push a model past its own demonstrators' quality — the policy is optimizing against a learned notion of 'better,' not memorizing a fixed set of 'best' answers — while SFT by construction cannot exceed what's in its training set.

Demo

This demo makes the imitation-ceiling concrete: an SFT-style loop can only reproduce labeled examples, while a preference-based loop can rank and prefer an unlabeled response that no demonstrator ever wrote.

LABELED_EXAMPLES = {
    "Explain gravity to a 10 year old": "Gravity is the pull that keeps you on the ground and makes things fall down."
}

def sft_response(prompt):
    # Imitation: can only return what's in the labeled set (or something
    # very close to it after training). Nothing outside this ceiling.
    return LABELED_EXAMPLES.get(prompt, "<no labeled example for this prompt>")

CANDIDATES = [
    "Gravity is a force.",
    "Gravity is the invisible pull between objects with mass -- it's why a dropped ball falls, why the moon orbits us, and why you feel heavier standing than floating.",
]

def preference_rank(candidates):
    # A reward model doesn't need a labeled "ideal" response for this exact
    # prompt -- it only needs to be ABLE TO COMPARE the candidates, which is
    # a much easier judgment than authoring a great answer from scratch.
    scores = {c: len(c.split()) + (10 if "why" in c else 0) for c in candidates}
    return sorted(candidates, key=lambda c: -scores[c])

print("SFT ceiling:", sft_response("Explain gravity to a 10 year old"))
print("Preference ranking (no labeled ideal needed):")
for i, c in enumerate(preference_rank(CANDIDATES)):
    print(f"  {i+1}. {c}")

Try it yourself

1. Add a third candidate that's clearly better than both and confirm preference_rank puts it first without ever having a labeled 'ideal' response for that prompt.
2. Change the scoring heuristic to penalize length instead of rewarding it, and observe how the ranking flips — this is the seed of reward hacking you'll meet formally in Module 3.
3. Prompt sft_response with a question that has no labeled example, and note what a real SFT-only model would do instead of returning the placeholder string.
4. Write down, in your own words, why ranking two draft emails is easier for a human reviewer than writing the 'perfect' email from scratch.

References

• Christiano et al. — Deep reinforcement learning from human preferences
• Ouyang et al. — Training language models to follow human feedback (InstructGPT)