The latency budget: where the milliseconds go

Why this matters

Humans notice conversational lag with brutal precision — psycholinguistics research on turn-taking in human conversation puts the average gap between one person stopping and the other starting at only about 200 milliseconds, and anything much past that starts to feel like a laggy phone call instead of a conversation. That number is the entire reason this course exists: it's the target your pipeline is competing against, and it forces you to treat every stage as a budget line item instead of 'however long the API takes.' A builder who understands the budget writes different code from day one — they stream instead of buffer, they pick the smaller model when it clears the bar, and they know which 100ms are worth spending money to shave off and which aren't. A builder who doesn't understand the budget ships something that works great in a demo and feels sluggish the moment a real network is involved.

Demo

This budget calculator lets you allocate a fixed total latency target across the stages of the loop and see what's left over — and what breaks the budget — before you've written a line of a real pipeline.

Try it yourself

1. Run it as-is and confirm the pipeline is over budget; note by how many milliseconds.
2. Cut llm_first_token to 120 (a smaller, faster model) and re-run — check whether that alone clears the 500ms target.
3. Add a new stage, tool_call, at 300ms and see how much it forces you to cut elsewhere to stay on budget.
4. Change TARGET_MS to 800 (a phone-call-quality bar instead of a face-to-face bar) and note which stages stop mattering.

References

• Levinson & Torreira — Timing in turn-taking (Frontiers in Psychology)
• OpenAI Realtime API guide — latency considerations

Chat about this lesson