Turn a base model into an aligned, reasoning assistant: SFT, reward modeling, PPO, DPO and its RL-free cousins, GRPO/RLVR, reasoning-model training, synthetic data pipelines, evaluation, and safety tuning.
Ten modules that build the complete post-training stack for large language models, in the order a working practitioner actually builds it. You start with the post-training landscape itself — what changes between a base model and an assistant, and why capability and alignment fail independently — then build SFT correctly (curation, chat templates, loss masking, packing, LoRA), train and stress-test a reward model, and run full PPO-based RLHF with all four of its moving models. From there you cover the RL-free preference family (DPO, IPO, KTO, ORPO) and the modern verifiable-reward family (GRPO and RLVR for math and code), specialize into training and evaluating reasoning models, and build the synthetic and preference data pipelines that feed every earlier stage. The final modules close the loop: evaluating post-training changes rigorously (capability vs alignment, reward-model health, contamination) and safety/alignment tuning (refusal calibration, Constitutional AI, RLAIF, red-teaming, and a full pre-ship checklist). Python-first throughout, built on Hugging Face TRL, PEFT, and transformers, with every technique shown as runnable code.
Built by Lakshya Kumar
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyComplete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Take a small open-weight base model and post-train it — SFT first, then either DPO or GRPO — so the resulting model measurably beats the base model on a target benchmark or reward metric of your choice (a capability benchmark subset, a task-specific verifiable reward, or a reward-model score). Submit the full pipeline (data, training configs, code), the before/after eval numbers, and a written analysis of which specific stage or hyperparameter change moved the needle most and which changes didn't help.
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
I'm taking a "Post-Training & RL for LLMs" course. It covers, in order: the post-training landscape (base vs instruct, capability vs alignment, the alignment tax); SFT done right (curation, chat templates, loss masking, packing, LoRA); reward modeling (preference data, Bradley-Terry, calibration, reward hacking, over-optimization); RLHF with PPO (the four models, GAE, the KL penalty, the clipped objective, instabilities); direct preference optimization (DPO's derivation, beta, IPO, KTO, ORPO); RL with verifiable rewards (GRPO, RLVR for math and code, writing verifiers); reasoning models (long chain-of-thought, self-consistency, distillation, process vs outcome reward); synthetic and preference data pipelines (generation, filtering, dedup, LLM-as-judge, self-instruct, rejection sampling); evaluating post-training (capability vs alignment benchmarks, release gates, contamination); and safety and alignment tuning (refusal calibration, Constitutional AI, RLAIF, red-teaming, jailbreak resistance). My context: 1. The model I want to post-train (size, base checkpoint, open-weight or hosted) is: [describe] 2. What I want it to do differently after post-training: [follow instructions better / prefer a certain style or safety posture / solve verifiable math or code tasks / reason step by step] 3. My current data situation: [no labeled data / some SFT examples / some preference pairs / a verifier I could write] 4. My compute budget: [a single consumer GPU / a small cloud GPU budget / a serious multi-GPU budget] Given that, answer: - Which module should I start with, and why, given what I already have? - Name the specific post-training path (SFT-only, DPO, GRPO/RLVR, or full PPO) you'd recommend for my goal and budget, using the decision framework from Module 1. - Name the single most likely mistake I'll make in my first attempt, based on my stated data and compute situation. - If I only have one week, what's the smallest version of this pipeline that would still teach me something real? - Name one thing this course will NOT solve for me, so I have the right expectations going in.
Train a reward model on real or synthetic preference data, held-out-accuracy-validate it, then deliberately construct an RL or best-of-N loop that exploits a specific weakness in it (length bias or another axis) to demonstrate reward hacking concretely. Then apply at least one mitigation (length normalization, ensembling, a held-out judge check) and show the exploit margin shrink. Submit the reward model, the demonstrated hack, and the before/after mitigation results.
Write and audit your own verifier for a math or code task, then run GRPO (group-relative advantage, no value model) against it on a small model across a meaningfully difficulty-varied prompt set. Report the pass-rate curve across training, the zero-variance-group fraction, and at least one genuine reward-shaping gap you found and fixed.
The DPO derivation behind all of Module 5.