Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
Every training or inference step is really three different resources fighting for the schedule at once: compute (how many floating-point operations the tensor cores can chew through per second), memory bandwidth (how fast data moves between HBM and the streaming multiprocessors), and communication (how fast gradients or activations move between GPUs over NVLink or the network). A kernel is bound by exactly one of these at a time, and the entire discipline of ML systems work is figuring out which one before you touch anything. Beginners reach for the fix that worked last time — bigger batch size, more workers, a fancier optimizer — without first asking which resource is actually saturated. If a kernel is memory-bound, adding more compute (a bigger matmul, more tensor-core throughput) buys you nothing, because the GPU is already sitting idle waiting for bytes to arrive. Learning to name the bottleneck before you name the fix is the single habit this entire course is built around.
The demo runs two kernels on the same GPU and reports achieved throughput against two different ceilings: TFLOP/s for a compute-heavy matmul, and GB/s for a memory-heavy elementwise op. The two numbers aren't comparable to each other, but each is comparable to your GPU's spec sheet — that comparison is the whole point.
nvidia-smi --query-gpu=name --format=csv prints) and compute what fraction of peak each kernel hit.n from 4096 down to 256 and re-run the matmul; explain why the achieved TFLOP/s drops even though it's 'the same kind' of operation.x + 1.0 to x * x + x and check whether the achieved GB/s changes even though there's more arithmetic per element.import torch, time
def bench(fn, warmup=10, iters=50):
for _ in range(warmup):
fn()
if torch.cuda.is_available():
torch.cuda.synchronize()
t0 = time.perf_counter()
for _ in range(iters):
fn()
if torch.cuda.is_available():
torch.cuda.synchronize()
return (time.perf_counter() - t0) / iters
device = "cuda" if torch.cuda.is_available() else "cpu"
# Compute-heavy: a big matmul spends most of its time on the tensor cores.
n = 4096
a = torch.randn(n, n, device=device, dtype=torch.float16)
b = torch.randn(n, n, device=device, dtype=torch.float16)
matmul_time = bench(lambda: a @ b)
matmul_flops = 2 * n ** 3
print(f"matmul: {matmul_flops / matmul_time / 1e12:.1f} TFLOP/s")
# Memory-heavy: an elementwise op moves a lot of bytes per FLOP done.
x = torch.randn(64_000_000, device=device, dtype=torch.float16)
add_time = bench(lambda: x + 1.0)
bytes_moved = x.numel() * x.element_size() * 2 # read x, write output
print(f"elementwise add: {bytes_moved / add_time / 1e9:.1f} GB/s")python3 main.py