How speech-to-text works
The inverse of this page: turning a spoken voice back into text the rest of the system can read.
Read it →How text-to-speech works.
How a computer turns written words into a natural, expressive voice you can talk with, not just read at.
This is the synthesis side of a voice loop: text in, a real-sounding voice out. Below is the plain version of how that happens, why it is hard, and where the field is heading. Every claim links to the paper it came from, so you can go straight to the original.
Jump to the source papers ↓Give the words timing, emphasis, and voice
Written input
“I can send it today.”
PronunciationRhythmEmphasisVoice quality
Synthesized audioIllustrative waveform
Turning written words into a spoken voice.
Text-to-speech, or TTS, converts text into an audio waveform of speech. The goal is not just to be understood. It is to sound natural: a voice a listener accepts as human, with the right rhythm and tone.
Inside, most systems do three jobs, and it helps to keep them in mind for the rest of the page. First, a front end reads the text and works out how it should be pronounced. Then an acoustic model decides what the speech should sound like, as a compact picture of the audio. Finally a vocoder turns that picture into the actual sound wave you hear.
The picture handed between the last two stages is usually a mel-spectrogram: a heatmap of which pitches are loud at each instant, frequency up the side and time across the bottom. Hold on to that word. It is the bridge in the diagram, and it is the thing the refraction figure above is drawing.
A short history, neutral and dated: early systems stitched together recorded clips, or ran a thin statistical voice. Both could be understood, but they sounded flat. Over the last decade neural TTS, which learns a voice from recordings, changed what synthetic speech sounds like, starting with WaveNet in 2016.
Convert text into an audible signal
- Text front end
Expand abbreviations and determine pronunciation.
- Acoustic representation
Model timing, pitch, and other sound features.
- Waveform generation
Create audio from the representation.
One sentence has many right readings.
Text underspecifies speech. A single line has countless valid deliveries, and a flat average of all of them sounds lifeless. The model has to choose one.
This one-to-many problem is the heart of the difficulty, and it is where prosody comes in: the rhythm, stress, and intonation of speech. Prosody is not decoration. It marks a question versus a statement, the new information versus the given, urgency, care, even sarcasm. Get it wrong and you do not just sound odd. You can change what the sentence means.
Three more things make it hard. Audio is huge and unforgiving: speech is tens of thousands of samples a second, and a small error becomes an audible buzz or a robotic timbre. Pronunciation is not in the spelling: "lead" the metal and "lead" the verb, names, "Dr.", "2026", "St." all have to be resolved before you can speak them. And for conversation it has to happen in real time, so the voice starts almost at once and keeps up with playback. That last pressure shapes the whole design, and the full pipeline budget is its own topic, handed to low-latency voice.
Delivery changes how a sentence is heard
Example input
You’re sure.
- Question
- A rising ending can request confirmation.
- Reassurance
- A steadier delivery can express confidence.
- Emphasis
- Stress on “sure” can contrast with uncertainty.
The modern pipeline, one stage at a time.
Follow the spine: front end, then the acoustic model, then the vocoder. Then the newer families that fold them together. Each named system links to its paper.
1The front end
First, work out the words.
Text normalization expands numbers and abbreviations into words you can actually say. Then grapheme-to-phoneme conversion turns spelling into the sounds to pronounce, and predicts where the phrase breaks fall. It is often a separate, smaller component, and it is where homographs, names, and dates get settled before anything is spoken.
Resolve written forms before pronunciation
| Written input | Spoken interpretation |
|---|---|
| Dr. Lee | Doctor Lee |
| 2026 | Twenty twenty-six, in this year context |
| $40 | Forty dollars, if USD is intended |
Pronunciation representation
Map the normalized words to the model’s pronunciation units.
2The acoustic model
Sketch the sound: text to mel-spectrogram.
Two families do this. The autoregressive, attention-based approach, set out by Tacotron 2, maps characters to a mel-spectrogram one frame at a time, then hands it to a neural vocoder: the canonical two-stage recipe. The non-autoregressive approach, set out by FastSpeech 2, predicts the whole spectrogram in parallel using a duration predictor, and conditions on explicit pitch and energy. It is faster, more stable with fewer skipped or repeated words, and it turns prosody into inputs a developer can steer.
An acoustic representation describes sound over time
- Horizontal direction
- Successive moments in the utterance.
- Vertical direction
- Frequency bands.
- Cell intensity
- Energy in that band at that time.
3The vocoder
Play it: mel-spectrogram to waveform.
The vocoder turns that picture into the sound wave. WaveNet generates the waveform one sample at a time, each conditioned on all the samples before it. It is widely credited with raising what synthetic speech could sound like, and was one of the early neural vocoders, but sample-by-sample generation is slow. HiFi-GAN takes a different route: adversarial training plus a model of the periodic structure of speech, producing high-fidelity audio that it reports as faster than real time. That speed is what makes streaming and conversational latency practical.
A vocoder turns acoustic features into samples
- Conditioning features
Receive the predicted acoustic representation.
- Waveform model
Generate the corresponding audio samples.
- Playback
Deliver the resulting sound, in chunks if supported.
4Folding the stages together
End to end, and the codec shift.
VITS collapses the two stages into one model trained end to end. Its stochastic duration predictor samples a fresh rhythm each time, so the same sentence is not spoken identically twice. More recently, VALL-E reframes the whole task as language modeling over discrete audio tokens from a neural codec. Given the text and a few seconds of reference audio as a prompt, it continues in that voice. That is zero-shot voice and style from a short sample, carrying over speaker identity, accent, and even the feel of the room.
Three ways prosody gets modeled: explicit predictors (FastSpeech 2) · stochastic variation (VITS) · in-context from a reference clip (VALL-E).
Condition new words on a reference voice
Reference audio
- Provides
- A short sample used to represent a voice.
New text
Example input
Thanks for the update. I’ll send the quote.
Conditioned audio
Synthesize the new text with the supported voice characteristics.
A conversation cannot wait for the full sentence.
At the synthesis level, the trick is to start speaking as the words arrive, rather than waiting for the whole utterance, and to keep time-to-first-audio low.
Parallel, non-autoregressive acoustic models paired with a faster-than-real-time vocoder let a system emit audio in chunks. There is a trade-off worth naming: autoregressive models can stream sample by sample but are slower, while parallel models trade a little look-ahead for speed. The full end-to-end latency budget, where speech recognition, a language model, and synthesis all share the clock, is its own subject, handed to low-latency voice, with the back-and-forth timing of who speaks when covered by turn detection.
Start playback from a usable first chunk
Generate full sentence
Generate all audio, Begin playback
Stream chunks
Generate first chunk, Play while later chunks are generated
Time →
What good synthesis makes possible.
Once a voice is natural, expressive, and fast, the uses open up.
-
Interfaces you talk with, not at
Natural, low-latency synthesis is what turns a voice agent from a script reader into something you can interrupt, correct, and have a back-and-forth with.
-
Delivery that fits the moment
A voice that can be warm, brisk, calm, or urgent as the moment needs, and that reads questions, lists, and emphasis correctly.
-
A voice from a short sample
Zero-shot synthesis carries a consistent voice across long content from seconds of reference audio, rather than hours in a studio.
-
Reach across languages and uses
Multilingual and multi-speaker voices, narration and audiobooks, accessibility and voice restoration, dubbing, and real-time spoken translation.
The research is moving fast.
A few directions are clear from the recent work.
Codec language models and generative audio. Larger models over discrete audio tokens are pushing zero-shot quality, in-context style, and speech editing, where you change a word without re-recording the line. Diffusion and flow-matching methods are being explored for high-fidelity, controllable synthesis and faster sampling.
Toward speech-to-speech and full-duplex. Systems that listen and speak in the same loop promise more natural turn-taking, overlap, and back-channels. Alongside that, finer control: steering emotion, pace, and emphasis with simple instructions, and holding a voice consistent over a long passage.
Responsibility and provenance. Because a voice can now be cloned from a short sample, the field is building consent norms, watermarking, and detection, so synthetic speech can be identified. This is the field maturing, stated as a field issue, not a claim about any one product.
Different research directions target different limits
| Direction | Focus |
|---|---|
| Codec models | Represent audio as sequences of learned codes. |
| Diffusion / flow models | Generate high-quality acoustic or waveform outputs. |
| Full-duplex systems | Listen and speak within one interaction. |
| Fine control | Control pace, style, and delivery. |
| Provenance | Identify origin and establish permitted voice use. |
A few things worth clearing up.
Short answers, in the same neutral spirit as the rest of the page.
Is this how AI REV Voice or Prism works?
No. This page explains the field in general, drawn from published research. It makes no claim about how our own systems are built, and the pipeline here should not be read as a description of Prism's internals.
What is a mel-spectrogram, in one line?
A heatmap of which pitches are loud at each instant, frequency up the side and time across the bottom. It is the compact picture of sound the acoustic model hands to the vocoder.
Can I use the models named here?
Whether a given system is available, and under what licence, is for the original authors to state. Follow the link to the source and check there.
Are the timeline and waveform figures real measurements?
No. They are illustrative, drawn to show the shape of the idea rather than measured numbers. For measured latency, follow the link to low-latency voice.
The voice is what carries the conversation.
The reason any of this matters to a seller is simple. Natural, expressive, low-latency synthesis is what separates a robotic readout from a voice you can actually talk with: one that briefs you on the drive, answers in the moment, and sounds like a colleague riding shotgun rather than a phone menu.
AI REV Voice runs on Prism, our own model, built for sales conversations. The outcome is the point: you talk, Prism talks back, and it feels like a conversation, not a command line. That is the whole promise behind Talk. Done.
The rest of the voice loop.
This page covered synthesis, the voice itself. Here is where the other parts of the loop are explained.
Low-latency voice
The full streaming loop and where the time goes: recognition, a language model, and synthesis sharing one clock.
Read it →Turn detection
Knowing when a speaker has finished, and how to handle being interrupted, so a back-and-forth feels human.
Read it →The papers, in full.
-
van den Oord et al., 2016. An early neural vocoder. Generating raw audio sample by sample yields markedly more natural speech than the systems before it.
arXiv:1609.03499 -
Shen et al., 2018. The canonical two-stage neural recipe, text to mel-spectrogram to neural vocoder, that most later systems build on.
arXiv:1712.05884 -
Ren et al., 2020. Non-autoregressive, parallel generation with a duration predictor and explicit pitch and energy control. The anchor for prosody and parallel synthesis.
arXiv:2006.04558 -
Kong et al., 2020. A GAN vocoder that reports high-fidelity audio faster than real time, which is what makes streaming and conversational latency practical.
arXiv:2010.05646 -
Kim et al., 2021. A single-stage, end-to-end model whose stochastic duration predictor gives natural rhythm variation, so the same line is not spoken identically twice.
arXiv:2106.06103 -
Wang et al., 2023. Reframes TTS as language modeling over discrete audio codec tokens, enabling zero-shot voice and style from a few seconds of reference audio.
arXiv:2301.02111
Talk. Done.
You have read how a voice is made. The faster way to understand it is to hear one. Hand Prism a phone number and have the conversation.
No app required, just your phone number.