Latency vs. Throughput: Why They Trade Off

Why this matters

Latency is how long one unit of work takes to finish; throughput is how much work finishes per unit of time across many units running together. Batching improves throughput because fixed overheads (kernel launches, memory reads that get reused across the batch) amortize over more work, but it necessarily raises the latency of any single item, because that item now waits for the rest of the batch to be ready. This tension is not a bug to be engineered away — it's a real dial you set deliberately. A training job cares almost entirely about throughput (bigger batches, more GPU-seconds of useful work per wall-clock second), while an interactive inference endpoint has a latency SLA it cannot violate no matter how good the throughput number looks. Learning to read this tradeoff as a curve, not a single number, is what lets you pick a batch size on purpose instead of copying whatever number a tutorial used.

Demo

The demo simulates a fixed per-batch overhead plus a per-item cost and sweeps batch size, printing both the average per-item latency and the aggregate throughput — the same curve shape you'll see again with real inference servers in Module 9.

Try it yourself

1. Run it and identify the batch size at which throughput roughly doubles from batch=1 but worst-case wait time also roughly doubles — that's the tradeoff, not a bug.
2. Raise fixed_overhead_ms to 20 (simulating a slower launch/dispatch path) and see how much larger a batch you need before amortized latency looks reasonable.
3. Add a max_latency_ms SLA parameter and find, by trial, the largest batch size that keeps worst_case_wait_ms under it.
4. Explain in a comment why training jobs are usually only sensitive to throughput while an interactive chat endpoint is sensitive to worst_case_wait_ms instead.

References

• NVIDIA Triton Inference Server — Batcher docs
• Horace He — Making Deep Learning Go Brrrr From First Principles