Project · Diagnose a real model's post-training recipe

Hints

• Qwen2.5, Llama 3, and Gemma 2 all publish reasonably detailed model cards — pick whichever you can actually download the base and instruct checkpoints for.
• If the card is thin on RL details (common for the smallest open releases), say so explicitly rather than inventing detail — 'the card does not disclose X' is a valid, honest finding.

Expected output

Model: Qwen2.5-0.5B (base) / Qwen2.5-0.5B-Instruct
Recipe: SFT + DPO (per model card, section "Post-training")
Prompt 1 (factual): "What is the capital of France?"
  base:     "...What is the capital of Germany? What is..."
  instruct: "The capital of France is Paris."
  classification: alignment gap (capability was present; presentation wasn't)
...
Recommendation: for a narrow customer-support fine-tune with 2,000 labeled
tickets and no preference data yet, start with SFT-only (LoRA) on top of
the Instruct checkpoint; revisit DPO only if tone consistency remains a
measured problem after SFT.

Stretch goals

• Repeat the comparison on a model whose card explicitly says RLAIF or Constitutional AI was used, and note anything qualitatively different about its refusals.
• Find a case where the base model actually outperforms the instruct model (e.g. raw completion quality on a creative-writing prompt) and explain why using the alignment-tax idea.
• Compute a rough compute-budget estimate (Task 7's cost model) for reproducing just the SFT stage of your chosen model at a 10x smaller data scale.