Backchannels
Telling "keep going" apart from "stop, let me talk" is still genuinely hard.
Turn detection in voice agents
VAD, endpointing, barge-in: how a voice agent knows when it is your turn to talk, and when it is its turn to answer.
An explainer of the field, written plainly. What turn detection is, why it is harder than "just wait for silence," how the modern systems work, and where the research is heading.
When should the agent speak?
You
“Send the revised quote…”
A brief pauseKeep listening
You
“…by Friday.”
The thought is completeReady to reply
Agent
“Friday. Got it.”
Two people trade turns without thinking. An agent has to do it on purpose.
When two people talk, they trade turns automatically. You finish a thought, a tiny gap passes, the other person answers. You pause to find a word and they wait. You jump in to correct something and they stop. None of that is written down anywhere, yet we get it right hundreds of times a call.
A voice agent has to do the same thing deliberately. Turn detection is the set of techniques that answer the one question a microphone cannot answer on its own: is it still your turn, or is it mine now. Get it right and talking to the agent feels like a conversation. Get it wrong and it either talks over you or leaves you sitting in dead air, like a walkie-talkie where you wait for the beep.
It sounds simple ("just wait for silence"), and that is exactly the trap. Silence is the weakest clue there is: a pause while you think looks identical to a pause because you are done. So the field moved from waiting on a silence timer to reading the cues people actually use, the melody of your voice, whether your sentence is grammatically finished, the rhythm of the back and forth, and from reacting after you stop to anticipating when you are about to.
The vocabulary you will meet everywhere.
Turn-taking is the coordination of who holds the floor and when the next person may start. Humans do it with sub-second gaps and very little overlap. Turn detection is the agent's version of that skill, and it breaks into a handful of named parts.
The whole page builds on one contrast. Half-duplex means strict take-turns: push-to-talk, one speaker at a time, wait for the beep. Full-duplex means always listening, even while speaking, the way people actually talk. Most of the progress in the field is the move away from the former, toward the latter. For the field map across all of these, the Skantze review is a strong place to start.
Different signals answer different questions
| Component | Question |
|---|---|
| Voice activity | Is speech present now? |
| Endpointing | Has this utterance ended? |
| Turn-taking | Is the person yielding the conversational floor? |
| Barge-in | Should the agent stop speaking? |
| Backchannel | Is this a brief acknowledgment? |
One speaker at a time, or overlapping streams
Half-duplex
- Your turn
The agent waits.
- Agent’s turn
You wait for the floor.
Full-duplex
- Agent is speaking
Incoming audio is still processed.
- You acknowledge or interrupt
The system decides whether to continue or yield.
The hard part is the middle.
The central tension is latency against interruption. A short silence timer makes the agent fast, but it cuts you off mid-sentence. A long timer stops the cutoffs, but adds laggy dead air. A fixed timer can never find the middle, because it has no idea why you went quiet.
A silence timer trades delay against interruption
| Policy | Potential cost |
|---|---|
| Short timer | Can interrupt a thinking pause. |
| Long timer | Can delay a response after a completed turn. |
| Learned decision | Uses richer cues, but still makes prediction errors. |
Silence is ambiguous. A thinking pause, a breath, a hesitation ("um") and a finished turn can all be the same length of quiet.
The real "I'm done" cues are multimodal. People signal it with falling pitch, final lengthening, grammatical completeness, gaze and breathing. Silence alone is a weak signal; the strong ones are in the voice and the words. The Skantze review catalogues them.
Backchannels look like turn-grabs. "Yeah" can mean I agree, continue, or it can mean stop, let me talk. Telling them apart is genuinely hard.
The agent must not interrupt itself. For barge-in to work, the system has to tell your voice apart from the echo of its own voice coming back through the line.
Real audio is hostile. Phone-band 8 kHz audio, packet loss, codec artifacts, road noise, hands-free cabins, crosstalk and a speaking rate that keeps changing.
Norms vary, and evaluation is slippery. Gap length and tolerance for overlap differ across people, languages and cultures. There is no single correct moment to respond, so metrics trade cutoff rate against latency. "Accuracy" here is a curve, not a point.
Five layers, classic to modern.
Each layer reads more of the signal than the one before it. The throughline is fusion: the strongest systems combine acoustic cues with the meaning of the words.
1VAD, the front gate
Is anyone speaking right now.
A small neural classifier labels each short frame of audio speech or non-speech in real time, resilient to noise and cheap enough to run on a CPU. It feeds a silence timer downstream. Silero VAD is a widely adopted open example of this front gate.
View the Silero VAD repo →Combine acoustic and linguistic evidence
- Speech activity
Detect speech versus non-speech.
- Prosody
Read timing, rhythm, and intonation cues.
- Language
Check whether the utterance projects more words.
- Turn decision
Combine evidence with the conversation state.
2Endpointing, beyond the timer
Learn the finish from the audio.
Classic endpointing is VAD plus a fixed trailing-silence threshold. Modern endpointing learns end of utterance from the acoustics, and end-to-end systems fold it straight into the recognizer, emitting an end-of-query token or running a parallel speech-detector head, which cuts the wait versus a static timer. See Bijwadia et al., 2022.
Read the endpointing paper →A learned endpoint can reduce unnecessary waiting
Fixed trailing timer
Speech, Wait for timer, Reply
Learned endpoint
Speech, Assess completion, Reply
Time →
3Semantic prediction
Read the words, not just the gap.
A language model reads the transcript so far and predicts whether the turn is pragmatically complete, even projecting where it will end. That lets the agent answer quickly after a complete sentence and keep waiting through a pause inside an unfinished one. This is the foundational case for predicting turns from meaning and syntax, shown by TurnGPT.
Read the TurnGPT paper →An unfinished phrase is evidence to wait
Example input
The renewal is in March, and the budget is…
- Acoustic state
- A pause follows.
- Linguistic state
- “The budget is” projects a value that has not been supplied.
Likely continuation
Keep listening for the missing amount.
4Continuous prediction
Anticipate, instead of react.
Rather than detecting an endpoint after the fact, a self-supervised model continuously predicts the near-future voice activity of both speakers from the raw audio. One representation yields end-of-turn, who-goes-next, backchannel and overlap prediction together. The shift is from detect to anticipate. See Voice Activity Projection.
Read the VAP paper →Predict near-future speech activity
Speaker A
Main turn
Speaker B
Brief acknowledgment, Next turn
Time →
- During the acknowledgment
- A short “mm-hm” may leave the floor with A.
- Near the end of A’s turn
- The model estimates whether A will continue or B will begin.
5Full-duplex foundation models
One model that listens while it speaks.
Rather than a pipeline of separate parts, a single speech-text model represents its own audio and the user's audio as parallel streams, always listening even while speaking. Barge-in and backchannels become native behaviors, not modules bolted on. Moshi is a full-duplex model of this kind, and reports roughly 160 ms theoretical and about 200 ms practical latency. That figure describes the Moshi paper, not any specific product.
Read the Moshi paper →Listen while producing the reply
- Agent speaks
Output audio is playing.
- Incoming audio continues
The system still processes the user’s microphone.
- Classify the interruption
Distinguish a backchannel from a request to take the floor.
- Respond
Continue, pause, or stop output as appropriate.
The throughline: fusion. Systems that pair acoustic cues (VAD, prosody) with linguistic cues (semantic completeness) hold up better than silence alone. The honest practical knobs are hangover and endpoint timeouts, a minimum-speech duration, echo cancellation for barge-in, and separate handling of backchannels.
A conversation, not a transaction.
When the turn-taking lands, the difference is something you feel rather than measure.
The feel of it
It waits when you think, answers when you are done.
Short, natural gaps. The agent holds while you pause to find a word, and replies promptly once you have actually finished. You can interrupt to redirect ("skip that, what is their renewal date?") and be heard right away, like a real phone call with no "please wait for the tone." A quiet "mm-hm" lands as listening, and your own "yeah" is not mistaken for a command to stop.
Where it pays off
It works where your hands are busy.
Genuinely hands-free and eyes-free use, on the drive or on the move, where you cannot press a button to say "I'm done." Fewer collisions and shorter gaps read as attentiveness and competence. Talking over someone reads as the opposite. That is the difference between a tool you adapt to and one that adapts to you.
Wait for the missing part of the instruction
Timer fires early
You
Move the meeting to…
Agent interrupts
What time?
You
…Thursday. I was still speaking.
Context-aware waiting
You
Move the meeting to…
The phrase is incomplete. Keep listening.
You
…Thursday at three.
Now confirm
Thursday at three. Shall I update it?
From waiting on silence to anticipating the turn.
The direction of travel across the field, and the problems still openly unsolved.
- Silence thresholds→Continuous, predictive models that anticipate the boundary
- Half-duplex pipelines→Unified full-duplex models that listen and speak at once
- Audio cues alone→Multimodal cues (gaze, video, breathing) for in-person and embedded or edge settings
- Endpointing in front of the model→Endpointing folded into the model, tighter across the loop
- Detecting turn cues→Generating them too, so the agent's own rhythm feels human
- Clean, single-language demos→Multilingual, multi-party, noise-resilient behavior proven in the field
Cross-cultural gaps
The right gap length and the tolerance for overlap differ by language and culture.
Honest evaluation
Metrics that reflect how a real conversation feels, not a single threshold on a curve.
The papers behind this page.
-
Silero VAD
A widely adopted, pre-trained voice activity detector that runs in real time on a CPU. The lightweight "is anyone speaking right now?" front gate that most stacks start from.
github.com/snakers4/silero-vad → -
TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog
Shows a language model can read the words so far and judge whether a turn is complete: predicting turns from meaning and syntax, not just silence.
arxiv.org/abs/2010.10874 → -
Voice Activity Projection: Self-supervised Learning of Turn-taking Events
A self-supervised model that continuously predicts the near-future speech of both speakers from raw audio, unifying end-of-turn, who-goes-next, backchannel and overlap into one anticipatory signal.
arxiv.org/abs/2205.09812 → -
Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systems
Folds endpointing directly into the speech recognizer, so end of utterance is learned from the audio instead of waiting on a fixed silence timer, reducing response latency.
arxiv.org/abs/2211.00786 → -
Turn-taking in Conversational Systems and Human-Robot Interaction: A Review
The field map: the multimodal cues humans use (prosody, syntax, gaze, breathing) and how systems model end-of-turn detection, interruptions and multi-party turns. A strong single overview.
sciencedirect.com · S088523082030111X → -
Moshi: a speech-text foundation model for real-time dialogue
A full-duplex speech-text model that represents its own and the user's audio as parallel streams, so it listens while it speaks and treats barge-in and backchannels as native behavior. Reports roughly 160 ms theoretical and about 200 ms practical latency.
arxiv.org/abs/2410.00037 →
The rest of the speech loop.
Turn detection sits inside a larger loop. These explainers cover the pieces around it.
Talk. Done.
All of this is the groundwork for one thing: talking to a co-pilot that feels like a conversation, not a walkie-talkie. To feel the difference, talk to Prism.
No app required, just your phone number.