Build voice agents that actually feel like a conversation. Streaming ASR and TTS, WebRTC and telephony transport, turn-taking and barge-in, native speech-to-speech models, latency engineering, function-calling on a live call, and shipping with real evals and monitoring.
Ten modules that build a complete realtime voice agent stack, in the order a working practitioner actually builds it. You start with the ASR-to-LLM-to-TTS loop and the sub-500ms latency budget that governs every design decision, then build real streaming speech-to-text and text-to-speech, wire up WebRTC and telephony transport, and implement genuine turn-taking — barge-in, VAD tuning, backchannels, endpointing — instead of the naive fixed-silence version most tutorials stop at. From there you work with native speech-to-speech models (OpenAI Realtime, Gemini Live) and compare them honestly against the assembled pipeline, do focused latency engineering under real concurrency, bridge into real phone calls over SIP and Twilio Media Streams with DTMF and compliance handled properly, build function-calling voice agents with confirmation gates and graceful human handoff, and close with the eval harness, production monitoring, and incident-response discipline a real launch needs. Python-first throughout, with WebRTC/browser code in JavaScript and telephony relay code in Go wherever a real equivalent exists, every technique shown as runnable code you can lift into your own system.
Built by Lakshya Kumar
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyComplete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Build and ship a complete realtime voice agent — reachable over the web (WebRTC/LiveKit) or over a real phone number (Twilio or equivalent) — that holds a natural conversation, correctly handles barge-in and turn-taking, calls at least two tools mid-conversation (one read-only, one gated behind an explicit confirmation), and meets a latency budget you state and measure (e.g. p95 time-to-first-audio under 700ms). Submit the working agent, a recording or transcript of at least 3 real test calls exercising tool use and at least one genuine interruption, and a written latency report showing your measured p95 against your stated budget.
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
I'm taking a "Voice & Realtime AI Agents" course. It covers, in order: the realtime voice stack (ASR-LLM-TTS loop, the sub-500ms latency budget, audio fundamentals); streaming speech-to-text (VAD, endpointing, confidence, noise, WER); streaming text-to-speech (SSML/prosody, voice cloning, first-byte latency, chunking, interruption); realtime transport (WebRTC vs WebSockets, LiveKit/Pipecat, jitter buffers, loss handling, SFU scaling); turn-taking and interruption (barge-in policy, VAD tuning, overlap, backchannels, context-aware endpointing); speech-to-speech models (OpenAI Realtime API, Gemini Live, vs the assembled pipeline); latency engineering (profiling under load, caching, region selection, percentiles, SLOs); telephony and channels (SIP, Twilio Media Streams, DTMF, call control, mulaw, compliance); function-calling voice agents (tools mid-call, memory, confirmation gates, handoff, error recovery); and evaluating and shipping voice (voice-specific evals, monitoring, cost, A/B testing, load testing, incident response). My context: 1. The voice agent I want to build is: [describe — web widget, phone line, both, and what it needs to do] 2. My current voice-AI experience: [none / I've called a TTS or ASR API once / I've built an assembled pipeline / I've used a Realtime API] 3. My biggest concern right now is: [latency / turn-taking feeling robotic / telephony integration / cost / I don't know where to start] 4. My stack: [language, whether I'm using LiveKit/Pipecat/a Realtime API, telephony provider if any] Given that, answer: - Which module should I start with, and why? - Name the first three things I should measure about my own pipeline before optimizing anything. - Given my biggest concern, name the single most relevant task in this course and what I should build from it first. - If I only had one weekend, what's the smallest version of my voice agent idea that would actually prove the concept? - Name one thing this course will NOT solve for me (e.g. a specific compliance question, or model quality itself), so I have the right expectations.
Take a voice pipeline (yours from this course, or another open one) and produce a full latency optimization pack: stage-by-stage profiling under at least 3 concurrency levels, identification of the highest-variance bottleneck with a concrete fix applied and measured, a TTS/response cache with a reported hit rate, and a written latency SLO with a genuinely measured baseline. Submit the before/after profiling data, the specific fix, and the SLO document.
Build a phone-reachable voice agent that replaces a traditional press-1-for-sales IVR: it must handle at least 3 intents via speech, accept DTMF as an equally valid fallback for the same intents with DTMF taking priority on conflict, detect voicemail versus human answer on at least one outbound test call, gate any outbound calling through a documented compliance check (consent, AI disclosure, calling window), and gracefully hand off to a human with session context attached when needed. Submit a real call log/recording, the compliance gate's test results, and the full call-flow design.
The session model, function calling, and interruption handling behind Module 6.