Low-latency voice pipelines
How a spoken computer answers inside the same short gap a person would, by running listen, think, and speak at the same time.
This is a plain-language tour of the field, with links to the work behind each idea. We explain how the budget gets met. We do not publish a number for any one product.
First audio plays while later chunks are still being generated and synthesized.
Talking to a person, the reply lands almost at once.
Across many languages, the gap between one speaker stopping and the next starting centers near a fifth of a second (Stivers et al., 2009). That small window is the bar a spoken computer has to clear. Miss it and the conversation feels like a walkie-talkie: you talk, you wait, it answers. Clear it and it feels like a colleague.
A voice assistant has a lot to do inside that window. It has to hear you and turn sound into text, think of a reply, and speak the reply back. The slow way runs those one after another, each waiting on the one before. A low-latency voice pipeline is the set of techniques that make the three overlap and stream: the system starts on your words before you finish the sentence, and starts answering before it has finished deciding what to say.
Three jobs, run together.
Listen, think, speak. Plus knowing the moment you actually stopped talking.
Those three are the obvious jobs. Underneath them sits a fourth that decides whether the rest even matters: knowing when you stopped, so the reply does not trample your sentence or leave a long silence. That timing question is its own subject, covered on turn detection.
The single number a listener feels is time to first sound: how long after you stop until you hear the first syllable of the reply. It matters more than how long the whole answer runs.
So there is a budget. The whole loop has to fit inside a window people can feel. Anchor it to how we actually take turns, near a fifth of a second, then to the industry framing for real time. As a broad rule of thumb, the industry conversation puts end-to-end targets in roughly the 300 ms to 500 ms range, with speech platforms aiming well under that. For a sourced figure, NVIDIA Riva frames real time as far under 300 ms, with a 150 ms target.
Time →
Time from the user’s turn ending to the first audible agent response.
The parts add up faster than the budget allows.
The enemy is serial waiting, not any single slow box.
Latency stacks. Network in, listen, think, speak, network out. Run one after another, the parts add up past the budget quickly. Shaving one stage is not enough when each still waits its turn.
The model is the long pole. A language model normally writes one token at a time, and each token needs a full pass over the network. Audio is unforgiving too: speech is tens of thousands of samples per second, generated step by step, the problem WaveNet framed.
Quality, latency, and cost pull against each other. A bigger model sounds smarter but answers slower. More compute per request buys speed but cuts how many conversations a server can hold at once.
You cannot wait for the end of the turn. If the system only starts thinking once you fully stop, the budget is already gone. It has to act on a half-finished sentence, and sometimes revise.
- Network in
Deliver the incoming audio.
- Recognize
Produce usable words.
- Generate
Produce a response.
- Synthesize and deliver
Create audio and send it to the listener.
Make the stages overlap, and make the model faster.
Each technique below has a source paper. The thread tying them together: nothing waits for the step before it to finish.
1Streaming speech to text
Hear the words as they land.
A recognizer built as a transducer (RNN-T) emits text as you speak, character by character, instead of waiting for the full clip. The running guess updates and firms up, which removes the wait-until-you-stop tax on the listen stage. (He et al., 2018.)
Time →
2Stream the words into the model
Start thinking before the sentence ends.
Those partial words feed the model before you finish, so the think stage is already warm by the time you stop. The model works on a prefix and extends it as more words arrive, rather than starting cold on a finished transcript.
| Audio progress | Available transcript |
|---|---|
| Partial 1 | Send the revised… |
| Partial 2 | Send the revised quote… |
| Final | Send the revised quote Friday. |
Update the candidate response as more context arrives.
3Speculative decoding
Guess ahead, then check in one pass.
A small, fast draft model proposes several tokens ahead. The big model verifies them in one parallel pass and keeps the run that matches. You get more tokens per step with the exact same output as the big model alone, and no retraining. Reported 2x to 3x faster decoding. (Leviathan et al., 2023.)
Draft tokens
Verified continuation
4The KV cache
Reuse the work already done.
A transformer can reuse the keys and values it already computed for earlier tokens, instead of recomputing the whole context each step. That cache is what makes streaming generation affordable. Managing it well, like paged memory, lets one server hold many live conversations at once. Related ideas, multi-query and grouped-query attention, shrink the cache further. (Kwon et al., 2023.)
| New output | Cached state reused | New state added |
|---|---|---|
| Token 1 | Prompt | Token 1 |
| Token 2 | Prompt + token 1 | Token 2 |
| Token 3 | Prompt + tokens 1–2 | Token 3 |
5Streaming text to speech
Speak before the sentence is finished.
The voice stage does not wait for the full reply text. It starts speaking the first words while the model is still writing the rest, playing one chunk of audio while generating the next, so spoken latency stays roughly constant instead of growing with sentence length. (Ma et al., 2019.) Modern streaming neural audio codecs push first-audio delay lower still.
Time →
6Pipelining ties it together
All three, at the same time.
Listen, think, and speak run overlapped, not in sequence. All three are in flight at once on different parts of the same utterance: the start of your sentence is being spoken back while the end is still being heard. This is the mental model to keep.
Time →
7The unified frontier
One model for all three.
Newer systems drop the hand-off between boxes and model speech to speech directly, listening and speaking at the same time (full-duplex), which removes the between-stage latency entirely. Moshi reports about 200 ms practical latency for this approach. (Défossez et al., 2024.)
- Incoming audio tokens
Represent the user’s speech.
- Joint speech model
Model the ongoing conversational streams.
- Outgoing audio tokens
Generate the assistant’s speech.
The payoff, in human terms.
When the reply lands inside the human gap, the experience changes shape.
Natural turn-taking
Replies that land inside the human gap, so it feels like talking, not transacting.
Barge-in and interruption
You can cut in mid-sentence and it stops and listens, because it is streaming both directions at once. That is a property of full-duplex designs.
No dead air
Short acknowledgements and an immediate first syllable keep the line feeling alive instead of frozen.
Hands-free, real-world use
Low latency is what makes a call while driving feel usable rather than stilted.
Help that arrives in time to use
During a live conversation, tight latency is the difference between an answer that arrives while the moment is open and one that arrives after it has passed.
Fewer hand-offs, more overlap.
The direction of travel is to collapse the boundaries the pipeline used to need.
End-to-end speech-to-speech models that fuse listen, think, and speak, trimming the hand-offs, as in Moshi and the production realtime voice APIs in that direction.
Full-duplex as the default: listening and speaking on parallel streams, with interruption and overlap treated as normal, not edge cases.
Streaming neural audio codecs, the tokenized-audio backbone, pushing first-audio delay lower, and speculative decoding becoming routine, with longer draft windows and self-drafting variants.
Real-time multilingual conversation and live translation as the same pipeline, multiplied across languages.
And the open hybrid question: how much of the work runs near the user versus in a data center. As a field, this is a genuine trade-off between latency, capability, and cost, with no single right answer.
Near the user
- Potential gain
- Less network travel.
- Constraint
- Device power, memory, and model capacity.
Data center
- Potential gain
- Larger shared compute resources.
- Constraint
- Network delay and service scheduling.
The work behind this page.
The streaming recognition reference (RNN-T): text emitted as you speak, the listen-without-waiting lever.
He, Sainath, Prabhavalkar, McGraw, Alvarez, et al. · 2018 (ICASSP 2019)The foundational speculative-decoding result: 2x to 3x faster generation with identical output and no retraining.
Leviathan, Kalman, Matias · 2023 (ICML)Explains the KV cache and how managing it like paged memory keeps many live conversations fast at once.
Kwon, Li, Zhuang, Sheng, Zheng, et al. · 2023 (SOSP)The streaming TTS idea: speak the first words while still generating the rest, keeping spoken latency roughly constant.
Ma, Pino, Cross, Puzon, Gu · 2019Fuses listen, think, and speak into one full-duplex streaming model; reports about 200 ms practical latency.
Défossez, Mazaré, Orsini, Royer, Kharitonov, Hadjeres, et al. (Kyutai) · 2024The human-side anchor: turn-taking gaps center near 200 ms across languages, the reason the sub-half-second budget exists.
Stivers, Enfield, Brown, Englert, Hayashi, Heinemann, Hoymann, Rossano, de Ruiter, Yoon, Levinson · 2009 (PNAS)Frames real time as far under 300 ms, with a 150 ms optimization target. Used here for the budget figures, not as a paper.
Names the hard part of audio plainly: tens of thousands of samples per second, generated step by step.
van den Oord et al. (DeepMind) · 2016Three pieces this page leans on.
This page owns the latency budget and the pipelining story. Each stage has its own explainer.
The questions this raises.
Four things people usually ask once the budget and the overlap make sense.
Why not just wait until the sentence ends?
Waiting until you fully stop spends most of the budget before the system even starts. By the time it begins to think, the human turn gap is already gone. Streaming lets it work on a half-finished sentence and revise if the ending changes things, so the first reply still lands on time.
Does low latency mean the answer is less accurate?
Not by itself. Speculative decoding, for one, returns the same output as the larger model, only sooner. The real tension is between model size, speed, and cost: a bigger model can sound smarter but answers slower, so systems balance the three rather than trade away accuracy for speed.
What is the single number that matters most?
Time to first sound: how long after you stop until you hear the first syllable of the reply. A listener feels that opening gap far more than how long the whole answer runs.
Is one fused model better than separate listen, think, and speak stages?
It is the direction the field is moving, because fusing the stages removes the hand-offs between them. Pipelined stages still work well, though, and many systems run them overlapped rather than fully fused. Both can land inside the budget.
A co-pilot you talk to has to answer like a person.
That responsiveness is a research problem this field has been steadily solving. Prism is our own model, built for sales conversations. The result you feel: a briefing on the drive that rides like a colleague, not a form read aloud.
No app required, just your phone number.