Real-time speech translation.
One person speaks in their language. Another reads or hears it in theirs, while the conversation is still happening.
Real-time speech translation turns spoken words in one language into text or speech in another, with very little delay. We walk it the way the field is built: what it is, why it is hard, how cascaded and end-to-end systems do it, what it makes possible, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
The three ideas to carry: cascaded versus end-to-end, the read-write policy, and the quality-latency tradeoff.
¿Puede enviar la cotización?
Can you send the quote?
Translating spoken words as they are spoken.
The field name is speech translation (ST). When it runs live, it is called simultaneous or streaming speech translation.
You already meet it: translated live captions in a meeting, a travel app that turns a spoken question into the local language, the running translation under an international broadcast, and multilingual customer support. In each case a continuous acoustic signal in one language goes in, and faithful text or speech in another language comes out, produced with low latency.
The field names three task shapes clearly. Speech-to-text translation (S2TT) takes audio in and gives translated text out. Speech-to-speech translation (S2ST) takes audio in and gives translated audio out. The live versions of each translate while the speaker is still talking. The plainest way to picture it is a human interpreter in a booth at the UN, listening and speaking at the same time, always a few words behind, sometimes anticipating where a sentence is going.
Speech to text
- Input
Spoken Spanish.
- Output
Written English translation.
Speech to speech
- Input
Spoken Spanish.
- Output
Spoken English translation.
Two hard problems, stacked, against a clock.
Recognizing messy speech is hard on its own. Translating it well is hard on its own. A live system has to do both before the sentence ends.
Mistakes stack up.
Recognition error and translation error compound. A small slip early, a misheard word, becomes a wrong sentence late. Where the system chains parts, errors accumulate across every handoff.
You cannot wait for the full sentence.
A live system has to translate from a partial input, and accept that more context may arrive and change the right answer. Waiting for the whole sentence would make a conversation feel broken.
Some languages end on the verb.
When a language places the verb or the negation at the end, a system reading left to right has to anticipate words it has not heard yet. The field calls this the implicit anticipation problem.
Quality versus latency.
Waiting longer gives a better translation but a worse conversation. Committing sooner feels responsive but risks errors. This tradeoff is the heart of the field, and a system has to pick a point on it.
Take a word back, or never.
Re-translation systems can rewrite an earlier guess as more audio arrives, which reads as flicker. Commit-only systems never take a word back, so they have to be more careful about each one.
Direct pairs are scarce.
Paired data, audio in one language matched to text or audio in another, is far rarer than plain transcripts or text-to-text pairs. For speech-to-speech, carrying tone, emphasis, and a speaker's identity across the boundary is harder than getting the words right.
Two ways to build it, then a rule for doing it live.
Chain known parts in a line, or train one model end to end. Either way, a live system needs a policy for when to wait and when to commit.
Cascade
- Recognize
Speech → source text.
- Translate
Source text → target text.
- Speak
Target text → speech.
End-to-end
- Source audio
Provide the speech input.
- Translation model
Produce target text or audio directly.
Two ways to build it: a line of parts, or one model end to end. Each handoff in the top flow is a place error can compound.
1The cascaded approach
Chain three known parts.
The classic build chains components in a line: speech-to-text recognizes the source audio, machine translation turns text in language A into text in language B, and an optional text-to-speech step speaks the result. Its strength is that each stage uses a mature, well-studied component with abundant training data. Its weakness is that errors compound across the handoffs and each stage adds latency.
We keep the recognition and synthesis stages shallow here, because each has a page of its own. The recognition stage gets the full treatment in Speech-to-text, and the synthesis stage in Text-to-speech. This page owns the translation and the do-it-live parts.
- Recognize
Spanish audio → Spanish transcript.
- Translate
Spanish transcript → English text.
- Synthesize
English text → English audio, if needed.
2The end-to-end approach
Train one model to translate the audio directly.
Instead of chaining parts, train a single model to map source audio directly to translated text (S2TT) or translated audio (S2ST), skipping the explicit source transcript. The early demonstration that this is even possible is Weiss et al. (2017), a single sequence-to-sequence network that translated Spanish speech to English text without transcribing the Spanish first.
The strength is fewer handoffs, less compounding, and lower latency potential. The weakness is that it needs the scarcer paired data and is harder to debug, because there is no intermediate transcript to inspect when something goes wrong.
¿Puede enviar la cotización?
Target-language text or speech, depending on the model.
3Direct speech-to-speech
Audio in, audio out, in one model.
Direct speech-to-speech goes from source audio straight to target audio. An early direct model is Translatotron (Jia et al., 2019), which showed audio-in, audio-out translation in a single model and could even carry a speaker's voice across the translation using a separately trained speaker encoder. Later systems output discrete acoustic units rather than spectrograms, which made the audio easier to model.
This is where expressiveness becomes its own goal: carrying tone, emphasis, and pauses across the language boundary, so the translated voice sounds like a person and not a readout.
- Meaning to preserve
- The request to send a quote.
- Language to change
- Spanish input → English output.
- Delivery to model
- Voice identity, rhythm, and expression where the system supports them.
4Doing it live: read-write policies
When to wait, and when to commit.
This is the defining mechanism. A simultaneous system needs a rule for every step: read more input, or write an output. The reference idea is the wait-k policy from Ma et al. (2019), STACL, where the system starts translating after k source words and stays k words behind. Holding that gap is what forces it to anticipate, since it is always committing before it has heard the rest.
A fixed wait-k is one option. Adaptive policies decide on the fly whether enough has been heard to commit the next word, holding quality steady while shaving the delay where the sentence allows it.
| Read source | Write target |
|---|---|
| ¿Puede | Wait |
| enviar | Wait |
| la | Can |
| cotización? | you… |
Conceptual wait-k schedule, not a word-for-word linguistic alignment.
5Streaming speech-to-text translation
Apply the read-write idea to audio.
The wait-k idea was first about text words. Streaming S2TT applies it to incoming audio. A streaming system like SimulSpeech (Ren et al., 2020) splits the audio in real time, often with a connectionist temporal classification segmenter, and translates each piece under a wait-k strategy, emitting translated text as the speaker talks.
Latency in this setting is measured carefully, with metrics like Average Lagging that capture how far behind the speaker the system runs. Any specific lag figure belongs to the system that published it, never to anyone reading this page.
Time →
6Re-translation versus commit-only
Rewrite the guess, or never take a word back.
Two live strategies sit at opposite ends. Re-translation re-runs the translation as more audio arrives and may overwrite earlier words, trading stability for accuracy. The display can flicker as a guess is corrected. Commit-only, the wait-k family, never revises a written word, trading flexibility for a stable, flicker-free line.
Neither is simply better. A live caption that flickers can be hard to read, while a commit-only line that locked too early can carry an error to the end. The right choice depends on whether the reader values stability or the latest best answer.
Re-translation
Can you send…
Can you forward the quote?
Commit-only
Can you → send → the quote?
7Unified multilingual and multimodal models
Many languages and tasks, one model.
Recent systems fold many languages and tasks into a single model. SeamlessM4T (Seamless Communication team, 2023) is one model covering speech-to-text, speech-to-speech, text-to-text, text-to-speech, and recognition across roughly 100 languages. A companion system, Seamless (2023), adds streaming, low-latency translation and works to carry expressive elements like tone and pauses across languages.
The direction is one model that does the whole job, in many languages, in both directions, rather than a different model per language pair and per task.
| Input | Requested output |
|---|---|
| Source speech | Translated text |
| Source speech | Translated speech |
| Source text | Translated text |
| Source text | Target-language speech |
8How we know it works
BLEU for quality, latency for delay, reported together.
Translation quality is scored with BLEU, which compares the system output against one or more human reference translations by the overlap of word sequences, or n-grams. Higher is better. For speech-to-speech, a common trick is ASR-BLEU: transcribe the translated audio, then score that text with BLEU. Latency is the other axis, measured with metrics like Average Lagging that capture how far behind the speaker the system runs.
The field reports quality and latency together, because either alone is misleading. An honest limit to state: BLEU rewards surface overlap, not meaning, so teams pair it with human judgment on whether the translation actually says the right thing. These are the field's yardsticks, not a product score, and no figure here is ours.
Quality
- Compare
- System output with suitable human references.
- Measure
- Use defined metrics and, when needed, human evaluation.
Delay
- Measure
- How far output lags behind the incoming speech.
- Report
- State the streaming policy and latency metric.
Translate as you speak, and a cross-language exchange becomes a conversation.
At the field level: live translated captions and subtitles, interpreter and travel apps, accessible multilingual meetings and broadcasts, support across languages, and work across borders.
For a sales co-pilot, the bridge is at the outcome level. When translation runs live and low latency, people talk back and forth naturally instead of waiting through a relay. The harder cases in the section above, partial input and faithful meaning, are exactly the ones that decide whether a cross-language call feels like a conversation.
Low latency feels like talking.
When the delay is small, a cross-language exchange flows back and forth instead of stalling through a relay. That responsiveness is the outcome readers already value across the site.
Faithful means meaning survives.
Names, numbers, and commitments need to land correctly in the other language. That is the same accuracy bar the quality section above describes.
Streaming means you just talk.
Translating from partial input is what lets a back-and-forth run hands-free, by phone, with no typing and no app required.
Built on the field, not a claim.
Everything here describes how the field works. We make no claim about which techniques power any one product. See Low-latency voice for the delay budget.
This explainer covers published research. To see how AI REV Voice helps with sales conversations, explore Prism.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
More languages, all four directions.
Larger unified models that cover more languages and all four task directions, speech and text both ways, in one system rather than a model per pair.
Carry the voice across.
Speech-to-speech that carries tone, emphasis, and a speaker's voice across the language boundary, so the translation sounds like a person.
Adaptive read-write.
Read-write policies that decide on the fly, holding quality steady while shaving latency where the sentence allows it.
Streaming closer to the audio.
Lower latency and lighter models, including streaming and on-device options for private, offline use closer to where the audio is.
Where paired data is scarce.
Better support for languages with little paired data, using self-supervised pretraining and mined or synthetic data.
Recognition and translation, closer together.
Speech-aware language models that fold recognition, translation, and generation closer together rather than running them in separate passes.
The papers behind this page.
- Weiss et al. (2017). Sequence-to-Sequence Models Can Directly Translate Foreign Speech. arXiv:1703.08581.
- Ma et al. (2019). STACL: Simultaneous Translation with Implicit Anticipation and Controllable Latency using Prefix-to-Prefix Framework (wait-k). arXiv:1810.08398.
- Jia et al. (2019). Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model (Translatotron). arXiv:1904.06037.
- Ren et al. (2020). SimulSpeech: End-to-End Simultaneous Speech to Text Translation. ACL Anthology 2020.acl-main.350.
- Seamless Communication team (2023). SeamlessM4T: Massively Multilingual and Multimodal Machine Translation. arXiv:2308.11596.
- Seamless Communication team (2023). Seamless: Multilingual Expressive and Streaming Speech Translation. arXiv:2312.05187.
- Orientation survey (2023). Recent Advances in Direct Speech-to-text Translation. arXiv:2306.11646.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
Translate as you speak, and a cross-language call feels like a conversation. See it on a real call.
No app required. Just your phone number.