Amdahl's Law and Why Speedups Don't Compound

Why this matters

Amdahl's Law is the uncomfortable arithmetic behind why a huge kernel-level speedup so often produces a disappointing end-to-end result: overall speedup is capped by the fraction of total time you actually sped up, not by how much faster that fraction got. If your fused kernel makes the compute portion of a step 3x faster but that portion was only 50% of step time to begin with, your best-case overall speedup is roughly 1.6x, not 3x — the other 50% (dataloader stalls, communication, host-device sync) is untouched and now dominates. This is why profiling-first (the previous task) and Amdahl's Law are really the same idea from two angles: profiling tells you which fraction is addressable, and Amdahl's Law tells you the ceiling on what optimizing that fraction alone can buy you. It's also why real optimization work is iterative — you fix the biggest bottleneck, watch a new one become the biggest, and repeat.

Demo

The demo computes Amdahl's speedup for a range of addressable fractions and per-fraction speedups, making the diminishing-returns table concrete instead of an abstract law.

Try it yourself

1. Run it and find the row where making the addressable fraction infinitely fast (the 'local inf' column) still caps overall speedup under 2x; note what fraction that requires.
2. Solve, by trial values of local_speedup, roughly how fast the addressable 60% portion needs to get to hit an overall 2x speedup.
3. Add a fraction of 0.99 to the sweep and compare how much more forgiving Amdahl's Law becomes once almost the whole step is addressable.
4. Relate one row of this table to the Module 1 profiling-first task: if a profile shows 30% comm-bound and you eliminate all communication, what's your overall speedup ceiling?

References

• Amdahl — Validity of the Single Processor Approach (1967)
• Horace He — Making Deep Learning Go Brrrr From First Principles