The Roofline Model

Why this matters

The roofline model plots the best performance a kernel can achieve as a function of its arithmetic intensity: FLOPs performed per byte moved from memory. Two flat 'roofs' bound every kernel — a memory roof set by HBM bandwidth, and a compute roof set by peak FLOP/s — and they cross at a single ridge point. Kernels to the left of the ridge are memory-bound no matter how fast your tensor cores are; kernels to the right are compute-bound no matter how fast your memory is. This single number, computed from nothing but the operation's shape, tells you the ceiling before you've profiled anything, and it explains a pattern you'll see constantly in this course: matmuls are usually compute-bound, but layernorm, softmax, dropout, and activation functions are almost always memory-bound, because they move a byte for nearly every FLOP they do. That's why the fix for those ops is never 'faster math' — it's fusion, which is the subject of Module 4.

Demo

The demo computes arithmetic intensity for a few real operations and classifies each against a ridge point derived from representative GPU specs, so you can see which ops land on which side before ever touching a profiler.

Try it yourself

1. Look up your own GPU's peak TFLOP/s and HBM bandwidth, recompute RIDGE_POINT, and check whether the matmul's classification changes.
2. Change the matmul size from n=4096 to n=64 and recompute its arithmetic intensity; explain why a tiny matmul can fall on the memory-bound side of the ridge.
3. Add a fourth op, an attention score matmul for seq_len=2048 and head_dim=128 (FLOPs = 2 * seq^2 * d), compute its bytes, and classify it.
4. Halve the byte count for the elementwise add (simulating fusing the read of x with the write of the output into one pass) and see how much its arithmetic intensity improves.

References

• Williams, Waterman & Patterson — Roofline: An Insightful Visual Performance Model
• NVIDIA Nsight Compute — Roofline Analysis