Realtime voice agents: a latency problem

What this lesson shows

Voice is a realtime loop on a strict latency budget: stream the transcript, start answering early, allow barge-in, and take turns, or it feels laggy and rude.

The question it opens with

Text chat can take its time. A voice that pauses for seconds before every reply, or talks over you, feels broken even with a perfect model.

The walkthrough, in the lesson's own words

• Flip the toggles. Get dead air under budget and let the user interrupt.
• Waiting for the user to fully finish feels tidy. Predict how it feels.
• Voice is a realtime, interruptible loop on a latency budget, not a straight line.
• Let the user cut in and the agent stops talking.
• Dead air is under budget and the user can interrupt. The turn feels like talking to a person, not waiting on a machine.
• Too much dead air. The serial pipeline makes the user wait in silence. Stream the transcript and start answering early to overlap the work.
• Snappy now, but the user still can't interrupt a long answer. Turn on barge-in so they can cut in.
• Off by default is the naive line: speech-to-text, then the model, then text-to-speech, one strictly after another, with no way to interrupt. Turn on streaming and answer-early so the steps overlap instead of stacking, then turn on barge-in so the user stays in control.
• The weak link is the speech-to-text step. Accents, background noise, and code-switching (flipping between Spanish and English mid-sentence) all raise transcription errors, and every wrong word flows into the model as if the user really said it. Stack that on the listen, think, speak round-trip and a voice agent has far less room for error than a chat box.
• Timings here are illustrative milliseconds, not measured numbers, but the trade-off is real.
• Waiting for the user to completely finish before the model starts feels cleaner. Does it make the conversation feel better or worse?
• Waiting to fully finish adds dead air. People expect sub-second turn-taking, so streaming the transcript and answering early is what feels human, even though the pipeline is doing more at once.
• A tool call (looking something up, hitting an API) adds latency you can't overlap away, because the answer depends on the result. Real voice agents mask it: the agent says a short filler like "let me check that" so the line never goes silent while the tool runs.
• Voice AI is just speech-to-text, then an LLM, then text-to-speech in a line.
• It's a realtime, interruptible loop on a strict latency budget: stream, answer early, allow barge-in, and take turns.
• The newest voice agents swap the whole pipeline for one speech-native model (speech-to-speech: audio in, audio out, no text in the middle). That cuts dead air even further, and it preserves prosody, the tone, pauses, and emotion a transcript throws away, so the agent can hear that you're frustrated and soften its reply. The trade: it costs far more per minute, and with no transcript in the middle it's harder to log, debug, and control. Plenty of production teams still run the pipeline you just tuned for exactly that reason.
• A voice assistant talks over you, or leaves a long awkward silence before answering. What is actually going wrong?
• A turn-taking and latency-budget failure, not a dumb model. The silence means the pipeline is serial instead of streaming, and talking over you means barge-in and end-of-turn detection are off. The model's intelligence is fine; the realtime loop around it is misbuilt.
• You can read a voice agent as a realtime loop on a latency budget: stream, answer early, allow barge-in, and take turns, instead of a straight speech-to-text to LLM to text-to-speech line.

Key takeaway

You turned a laggy, robotic exchange into a responsive one by streaming, answering early, and allowing interruptions, all on a fixed latency budget.

What you can do after this lesson

You can explain why realtime voice is a systems problem: streaming, a latency budget, turn-taking, and barge-in, not just speech-to-text plus an LLM plus text-to-speech.

This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.