Stop guessing at GPU performance and start measuring it. Roofline models, real profilers, Triton kernels, memory optimization, data and model parallelism, multi-node training, inference serving, and the scaling-law math behind planning a real training run.
Ten modules that build the full GPU performance and distributed-training stack a working ML systems engineer actually needs, in the order they're used in practice. You start with the mental models — the roofline model, arithmetic intensity, and a profiling-first mindset — then learn the GPU hardware itself (SIMT execution, the memory hierarchy, tensor cores) and the real profilers (torch.profiler, Nsight Systems, Nsight Compute) that turn theory into measured numbers. From there you write real Triton kernels and just enough CUDA to read one, apply the memory-optimization toolkit (mixed precision, checkpointing, offload), and scale out — first data parallelism, then ZeRO/FSDP sharding, tensor and pipeline parallelism, and multi-node training with real torchrun/SLURM launch configs. The final modules cover inference-specific optimization (the KV cache, continuous batching, PagedAttention, speculative decoding) and the scaling-law math for planning and budgeting a real training run. Every technique is shown as runnable Python/PyTorch code, real Triton kernels, or real launch commands you can lift directly into your own system.
Built by Lakshya Kumar
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyComplete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Take a real training or inference workload you can run, profile it end-to-end using Module 3's workflow, identify the dominant bottleneck, and apply techniques from this course to speed it up by a target factor — at least 2x tokens/sec for a training workload, or 2x MFU/throughput for an inference workload. Submit the before/after trace, the specific fix or fixes applied, and a written diagnosis tracing the speedup to a specific bottleneck category (compute, memory, latency, or communication).
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
I'm taking an "ML Systems: GPU Performance & Distributed Training" course. It covers, in order: the roofline model, arithmetic intensity, and a profiling-first mindset; GPU fundamentals (SIMT execution, the memory hierarchy, tensor cores, occupancy); real profilers (torch.profiler, Nsight Systems, Nsight Compute) and reading a trace; writing Triton kernels (elementwise fusion, softmax, matmul) plus just enough CUDA to read one, and the flash-attention online-softmax algorithm; memory optimization (mixed precision, activation checkpointing, gradient accumulation, CPU/NVMe offload); data parallelism and DDP's all-reduce/bucketing/overlap; sharded training (ZeRO stages, FSDP, tensor and pipeline parallelism, Megatron-style 3D parallelism); multi-node training (interconnects, NCCL collectives, torchrun/SLURM, fault tolerance, checkpointing); inference optimization (the KV cache, continuous batching, PagedAttention, quantization, speculative decoding); and scaling laws, MFU, and training-cost planning. My context: 1. My workload is: [training a model from scratch / fine-tuning / serving an LLM for inference / not sure yet] 2. My hardware access: [free Colab/Kaggle GPU / a single cloud GPU / multiple GPUs on one node / a multi-node cluster] 3. My current bottleneck (if I know it): [slow training steps / OOM errors / poor multi-GPU scaling / slow/expensive inference / I haven't profiled yet] 4. My framework: [PyTorch version, and whether I already use DeepSpeed, FSDP, vLLM, or similar] Given that, answer: - Which module should I start with, and why, given my current bottleneck? - What's the single fastest diagnostic (from Module 1 or 3) I should run first to confirm what's actually limiting me? - Name one technique from this course that's very likely to give me the biggest win for my specific workload and hardware. - If I only have a free-tier GPU, which hands-on tasks in this course will I need to adapt or simulate rather than run for real? - Name one thing this course will NOT solve for me (e.g. it won't pick my model architecture or dataset), so I have the right expectations.
Pick a real fusion opportunity in an actual model — not a toy example — implement it as a Triton kernel, verify its numerical correctness against the eager PyTorch reference, and benchmark it against both the eager baseline and torch.compile at production-realistic shapes. Submit the kernel, the verification results, and the benchmark with a defensible ship-or-don't-ship verdict per Module 4's decision rule.
Take a real training job and scale it from 1 GPU to at least 4 (or a believable, clearly-labeled simulated equivalent), measuring throughput and scaling efficiency at each step. Profile the multi-GPU run to attribute any inefficiency to a specific cause — communication, bucketing, or per-GPU batch size — and apply the correct learning-rate scaling rule for the resulting effective batch size. Submit the throughput table, the profiler evidence, and the scaling-efficiency diagnosis.
The FLOPs/bytes/KV-cache arithmetic behind Modules 1 and 9.