The Three-Stage Pipeline: SFT, Reward Model, RL

Why this matters

Almost every modern post-training recipe — InstructGPT, Llama, Claude's public description of its own process — decomposes into the same three stages in the same order, and knowing why that order is load-bearing saves you from a lot of wasted experiments. Supervised fine-tuning (SFT) comes first because you need a model that already produces roughly the right shape of answer before you can meaningfully rank two answers against each other; without it, a reward model has nothing but noise to compare. Reward modeling comes second because reinforcement learning needs a scalar signal to optimize, and humans can reliably rank two responses ('which is better') far more consistently than they can assign an absolute quality score to one. RL comes last because it's the most expensive, least stable stage, and you only want to spend that budget refining a policy that's already close, not searching from scratch. Newer recipes (DPO, GRPO) compress or skip stages, but you can't understand why those simplifications work, or what they trade away, without first understanding the three-stage baseline they're simplifying.

Demo

The sketch below is not a real training run — it's the shape of the pipeline as a sequence of function calls, so you can see what each stage consumes and produces before we build any of them for real in later modules.

# Pipeline shape (illustrative — each function is a stub you'll build for
# real in Modules 2, 3, 4/6).

def sft(base_model, instruction_pairs):
    """Stage 1: train on (prompt, ideal_response) pairs with next-token loss,
    masked so only response tokens contribute. Produces a policy that
    already sounds like an assistant."""
    return "sft_policy"

def train_reward_model(sft_policy, preference_pairs):
    """Stage 2: train a scalar head on (prompt, chosen, rejected) triples
    using a Bradley-Terry loss. Produces a scorer, not a generator."""
    return "reward_model"

def rlhf(sft_policy, reward_model, reference_model, prompts):
    """Stage 3: sample from the policy, score with the reward model, and
    update the policy to raise its expected reward while a KL penalty
    keeps it near the reference (usually the SFT policy)."""
    return "rl_policy"

policy = sft(base_model="Qwen2.5-0.5B", instruction_pairs="sft_dataset")
rm = train_reward_model(policy, preference_pairs="pref_dataset")
final_policy = rlhf(policy, rm, reference_model=policy, prompts="rl_prompts")

# Each stage's OUTPUT is the next stage's INPUT — that dependency chain is
# why you can't skip straight to RL from a base model and expect it to work.

Try it yourself

1. Rewrite the stub pipeline so rlhf takes a DPO-style dataset instead of a reward model argument, and note which stage this eliminates entirely.
2. List, for each stage, whether its main cost driver is data-labeling, compute, or engineering complexity — you'll use this to reason about tradeoffs in Modules 4-6.
3. Identify which stage's output the reference model in stage 3 is usually copied from, and explain in one sentence why the reference has to be frozen.
4. Sketch what would go wrong if you trained the reward model on preference pairs generated by the base model instead of the SFT policy.

References

• Ouyang et al. — Training language models to follow human feedback (InstructGPT)
• Nathan Lambert — The RLHF Book