How speech-to-text works.
A messy call goes in. Clean, structured text comes out. Here is the field that makes that happen, in plain language.
Speech-to-text, also called automatic speech recognition or ASR, turns spoken audio into written words. We walk it the way the field is built: what it is, why it is hard, how a modern system does it, what it makes possible, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
“Send the revised quote Friday.”
Turning spoken audio into written words.
The field name is automatic speech recognition (ASR). Speech-to-text is the everyday name for the same job.
You already meet it everywhere: dictation on your phone, live captions on a video, the transcript of a voicemail or a meeting, the voice assistant on your counter, and search over recorded calls. In each case the work is the same. A continuous, messy sound signal goes in, and discrete, structured text comes out, often with punctuation, timestamps, and speaker labels attached.
The plainest way to picture it: a colleague typing out what you say, word by word, as you say it. The rest of this page is about how a machine does that job well, and why doing it well is harder than it sounds.
- Audio
A continuous signal with speech, pauses, and noise.
- Recognition
Estimate the spoken word sequence.
- Additional processing
Add punctuation, timing, or speaker attribution when supported.
The signal is messy in every direction at once.
Speech is not a tidy stream of separated words. It is a noisy, continuous signal that shifts with every speaker and every room.
No gaps between words.
Real speech runs together. Sounds blur into their neighbors, an effect called co-articulation, and pace and emphasis change constantly. The system has to find word boundaries that were never actually there in the audio.
Everyone sounds different.
Accents, dialects, pitch, and speaking style all vary, and so do disfluencies: the ums, the false starts, the self-corrections. The same sentence is a different signal in two different mouths.
The world is loud.
Background noise, road noise in a car at speed, narrow phone-band audio, reverberation, music, and crosstalk all distort the signal before it ever reaches the model.
People talk over each other.
When two people speak at once, the system has to separate the voices and attribute each word to the right person. That sets up speaker diarization and overlap handling later on this page.
Some words only context can settle.
Homophones (their, there, they're), names, numbers, currency, and domain jargon often sound identical. Only the surrounding context resolves which word was meant.
Live means no waiting.
A live system has to produce text with low latency and judge when a turn has ended, called endpointing, without waiting for the whole utterance to finish. Even scoring "close but not exact" is its own problem, which is why the field settled on word error rate, further down.
A modern system listens in stages.
Slice the sound, recognize the units, assemble the words, make it readable. Newer systems do this while you are still talking.
- 01Listen
Sample the incoming audio.
- 02Represent
Compute model input features.
- 03Recognize
Score likely speech units.
- 04Decode
Produce a word sequence.
- 05Format
Add supported text structure.
The pipeline the NVIDIA guide lays out: four jobs in a row, each doing one thing.
1From sound to features, then to sound units
Listen, then recognize.
First the waveform is sampled and turned into a spectrogram, usually mel features, using a short-time Fourier transform over small overlapping windows. The result is a sequence of feature frames, a few hundred per second.
Then a neural acoustic model maps those frames to probabilities over sound units, characters, or subword tokens. Two reference points: the Conformer (Gulati et al., 2020) pairs convolution for local sound detail with self-attention for long-range context, and wav2vec 2.0 (Baevski et al., 2020) pretrains on large amounts of unlabeled audio first, so the model learns the shape of speech before it ever sees a transcript. That sharply cuts how much labeled data is needed.
| Frame | Candidate 1 | Candidate 2 |
|---|---|---|
| t | /s/: 0.71 | /f/: 0.18 |
| t + 1 | /e/: 0.64 | /i/: 0.22 |
Use the acoustic evidence and the model’s sequence constraints.
2From probabilities to words
Decode, then make it readable.
A decoder, greedy or beam search, walks those probabilities into a word sequence. An optional language model, n-gram or neural, nudges it toward plausible phrasing, and a punctuation and capitalization step makes the text read cleanly.
Modern systems often fold these stages into a single network. Three families show up: CTC, the RNN-Transducer, and attention encoder-decoders. The RNN-Transducer (Graves, 2012) is built so output can be emitted as audio arrives, feeding each predicted symbol back through a prediction network. Whisper (Radford et al., 2022) is an attention encoder-decoder trained on a very large, weakly labeled, multilingual corpus, which buys resilience to noise and accents without task-specific tuning.
send the revised quote friday
Send the revised quote Friday.
3Streaming and partial hypotheses
Show a best guess, then commit.
A streaming, or online, system reads audio in small chunks and emits interim results, called partial hypotheses, that can change as more context arrives. Then it commits a final line. That is the contrast with batch or offline recognition, which waits for the whole clip before answering. The production account of running this in real time, end to end, is He et al. (2018).
Streaming recognition balances speed with context. Smaller chunks can produce words sooner, while more context can help resolve an uncertain phrase. The useful measure is the delay the listener experiences across the whole pipeline, not only the time spent running the model.
| Status | Visible text |
|---|---|
| Partial | send the… |
| Partial | send the revised… |
| Final | Send the revised quote Friday. |
4Endpointing
Knowing when you have finished a thought.
Endpointing is deciding when an utterance or a turn has ended, so the system can finalize the text and, in a voice agent, respond. We keep it light here, because it has a page of its own. Voice-activity detection, endpointing, and barge-in get the full treatment in Turn detection.
Send the quote to…
Speech activity
The microphone currently contains silence.
Turn meaning
The sentence still projects a recipient.
Combine timing with acoustic and linguistic evidence.
5Speaker diarization
Who spoke when.
Diarization labels each stretch of audio with a speaker, so one transcript becomes a speaker-attributed transcript. Two approaches: the classic pipeline of segmentation, speaker embeddings, and clustering, and end-to-end neural diarization, or EEND, which learns to label speakers directly and can handle overlapping speech, as in Fujita et al. (2019). Open building blocks exist for readers who want to try it, such as pyannote.audio (Bredin et al., 2019).
| Time | Speaker | Text |
|---|---|---|
| 00:00–00:04 | A | I can send the quote Friday. |
| 00:04–00:07 | B | Friday works. Include Marcus. |
6How we know it works
Word error rate, the field's yardstick.
The field measures itself with one plain number, word error rate. Align the system output to a human reference with edit distance, count the substituted, inserted, and deleted words, and divide by the number of reference words. Lower is better, and zero is perfect.
Honest limits are worth stating. Word error rate weights every word equally, so a wrong price and a wrong "the" count the same. Casing and punctuation are often scored separately. Domain terms and names matter more than the raw number suggests, which is why teams pair the metric with review on the words that actually count.
send the revised quote Friday
send the revise that quote
| Error | Example |
|---|---|
| Substitution | revised → revise |
| Insertion | that |
| Deletion | Friday |
WER = (1 + 1 + 1) / 5 = 60%Good speech-to-text is raw material for everything built on top.
At the field level: captions and accessibility, searchable call and meeting archives, voice interfaces and voice agents, and analytics over what was actually said.
For a sales co-pilot, the bridge is at the outcome level. Get the words right, get the speakers right, and get them fast, and a co-pilot can turn the call into clean, complete notes and a back-and-forth you can just talk to. The harder cases in the section above are exactly the ones that decide whether a call from the car becomes a usable note.
Accurate words become clean notes.
The better the words come out, the more reliably a co-pilot turns a call into a complete note in the CRM, with nothing missing. See After-meeting notes.
Speaker labels keep the record straight.
What the rep committed to and what the client asked for stay separate, so managers and RevOps get a record they can trust.
Streaming means you just talk.
Partial hypotheses and fast turn-taking are what make a spoken back-and-forth feel natural: hands-free on the drive, by phone, no typing, no app required, just your phone number.
Holds up in the real world.
Road noise, accents, and narrow phone audio are the hard cases from above. Strong speech-to-text is why those calls still become a usable record.
This explainer covers published research. To see how AI REV Voice helps with sales conversations, explore Prism.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
One model, many languages and tasks.
Larger multilingual, multitask models that cover many languages and jobs at once, and hold up better against noise and accents.
Faster and lighter at once.
Streaming that is both quicker and smaller, including edge and low-footprint options for low-latency recognition closer to where the audio is.
Transcription and understanding together.
Tighter coupling of transcription and understanding, so speech-aware language models produce words and meaning together rather than in separate passes.
Overlap, code-switching, personalization.
Better handling of people talking over each other, switching languages mid-sentence, and the names and jargon specific to a team.
Beyond a single number.
Evaluation that moves past raw word error rate toward semantic and entity-level accuracy: did it get the price, the date, and the name right?
The papers behind this page.
- Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.
- Gulati et al. (2020). Conformer: Convolution-augmented Transformer for Speech Recognition. arXiv:2005.08100.
- Graves (2012). Sequence Transduction with Recurrent Neural Networks (RNN-Transducer). arXiv:1211.3711.
- Baevski et al. (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. arXiv:2006.11477.
- He et al. (2018). Streaming End-to-end Speech Recognition for Mobile Devices. arXiv:1811.06621.
- Fujita et al. (2019). End-to-End Neural Speaker Diarization with Self-attention. arXiv:1909.06247.
- Bredin et al. (2019). pyannote.audio: neural building blocks for speaker diarization. arXiv:1911.01255.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
That is the whole field in one promise: you talk, and clean, complete notes are waiting. See it on a real call.
No app required. Just your phone number.