Where the Compute Actually Goes (FLOPs)

Why this matters

To optimize serving you must know where the work is. A transformer forward pass spends its FLOPs on the attention computation and the big feed-forward (MLP) matmuls, scaled by model size and sequence length. Knowing the rough cost breakdown — that compute scales with parameters and (for attention) with sequence length squared — tells you why bigger models and longer contexts cost more, and where techniques like quantization (shrink the weights) and attention optimizations (tame the n²) pay off. This is the quantitative intuition behind every later optimization.

Demo

The demo estimates FLOPs per token (~2 × parameters for the matmuls) and shows attention's sequence-length dependence, giving you a back-of-envelope for why a 70B model is ~10× a 7B and why long contexts strain attention.

Try it yourself

1. Compare FLOPs/token for a 7B vs. 70B model and confirm it scales ~linearly with parameters.
2. Increase seq_len and watch attention's share of the work grow — the motivation for attention optimizations.
3. Connect: halving the weights' precision (quantization) roughly halves the matmul cost/memory.
4. Reason about why decode (1 new token, but full weights read) is memory-bound despite low FLOPs — setup for Module 3.

References

• Kaplan et al. — Scaling laws (FLOP estimates)
• NVIDIA — LLM inference optimization

Chat about this lesson