Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
Budgeting a post-training project requires knowing that the three stages have wildly different cost profiles, and the difference isn't really about parameter count — it's about what dominates the wall-clock time. SFT is a standard supervised training loop: one forward and backward pass per batch, and it's the cheapest stage by a wide margin. Reward modeling adds a second, usually smaller model, but training it is still a conventional supervised loop over a fixed preference dataset. RL (PPO in particular) is the expensive one because each training step requires generating full responses token-by-token from the current policy before you can compute anything — autoregressive sampling is slow, and you typically need many rollouts per prompt — so the same GPU-hour buys you far less RL progress than SFT progress. This is precisely the economic argument behind DPO and GRPO's popularity: DPO removes the reward model and the rollout-heavy PPO loop entirely, and GRPO removes the separate value model, both directly attacking the cost structure you're about to compute below.
A back-of-envelope cost model below turns 'RL is expensive' into an actual number, so you can see where the multiplier comes from instead of just being told to expect it.
def sft_cost(n_examples, tokens_per_example, cost_per_1k_tokens=0.002):
total_tokens = n_examples * tokens_per_example
return (total_tokens / 1000) * cost_per_1k_tokens # one fwd+bwd pass each
def reward_model_cost(n_pairs, tokens_per_pair, cost_per_1k_tokens=0.002):
total_tokens = n_pairs * tokens_per_pair * 2 # chosen + rejected
return (total_tokens / 1000) * cost_per_1k_tokens
def ppo_cost(n_prompts, rollouts_per_prompt, gen_tokens, cost_per_1k_tokens=0.002,
rl_multiplier=4):
# rl_multiplier approximates the extra fwd/bwd passes for value + reward
# scoring + the policy update on top of generation itself.
gen_tokens_total = n_prompts * rollouts_per_prompt * gen_tokens
return (gen_tokens_total / 1000) * cost_per_1k_tokens * rl_multiplier
sft = sft_cost(n_examples=50_000, tokens_per_example=300)
rm = reward_model_cost(n_pairs=20_000, tokens_per_pair=200)
ppo = ppo_cost(n_prompts=20_000, rollouts_per_prompt=4, gen_tokens=256)
print(f"SFT: ${sft:,.2f}")
print(f"Reward model: ${rm:,.2f}")
print(f"PPO: ${ppo:,.2f}")
print(f"PPO / SFT ratio: {ppo/sft:.1f}x")python3 main.pyrollouts_per_prompt from 4 to 16 (a realistic value for better advantage estimates) and see how much the PPO/SFT ratio grows.rl_multiplier to 1 (pretending RL were as cheap as one forward pass) to isolate how much of the cost is generation itself versus the extra bookkeeping models.reward_model_cost's shape (no rollouts needed) and compare it directly against the PPO number.