Speaker diarization: who spoke when.
One mixed recording goes in. A transcript where every line is attributed to a speaker comes out. Here is the field that makes that happen, in plain language.
Speaker diarization answers one plain question about an audio file: who spoke, and when. It does not need your name, and it does not transcribe the words. It separates one stream into consistent speakers and labels them, Speaker A, Speaker B, Speaker C, with the right boundaries in time. We walk it the way the field is built: what it is, why it is hard, how a modern system does it, what it makes possible, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
The three ideas to carry: speaker embeddings, clustering, and end-to-end neural diarization.
Splitting one stream into who spoke when.
Diarization partitions an audio stream into segments and labels each one with a consistent, anonymous speaker tag.
A recording of a meeting or a call is just one stream of sound. Two, three, sometimes five voices are mixed together on the same track, taking turns, talking over each other, going quiet and coming back. A person can follow it: that was the buyer, that was the rep, the new voice is someone they just looped in. Diarization teaches a machine to do the same thing.
Three jobs get conflated constantly, so it helps to draw the boundaries cleanly. Speech to text, also called ASR, says what words were said, a separate page and a separate step. Diarization says who spoke and when, using anonymous, relative labels. Speaker recognition matches a voice to a named, enrolled person, a further step that diarization does not require. Pair diarization with a transcript and every line gets attributed to the voice that said it. That is the difference between a wall of text and a readable record of a conversation.
| Task | Example output |
|---|---|
| Transcription | “Friday works.” |
| Diarization | Speaker B · 00:08–00:09 |
| Identification | Speaker B is Priya, if established by evidence. |
"Just tell the voices apart" is the trap.
It sounds easy. Voices overlap, people sound different on a phone than in a room, and you do not know in advance how many speakers there will be.
The count is an output, not an input.
You rarely know in advance how many people are on a call. Guess wrong and the whole partition is wrong. Handling an unknown, variable number of speakers, two to five or more, is a core difficulty.
People talk over each other.
Real conversations have crosstalk, interruptions, and backchannels. A method that assumes one speaker per moment cannot label two voices at once. This is the limitation that motivated end-to-end methods.
A quick "yeah" gives little to work with.
Short turns and fast switches leave very little audio to fingerprint a voice from, and a speaker change can land mid-second. The boundaries are genuinely hard to place.
Same person shifts, different people clash.
The same person sounds different across a call: emotion, distance from the mic, phone versus room. Different people can sound alike. The embedding has to be robust to the first and sensitive to the second.
The signal arrives degraded.
Phone-band 8 kHz audio, packet loss, codec artifacts, reverberation, far-field room mics, and background noise all distort the voices before any model sees them.
The labels are arbitrary.
"Speaker A" and "Speaker B" are arbitrary tags, so a model has to be scored in a way that does not care which tag is which. This is why end-to-end systems use permutation-invariant training objectives, and why the headline metric, diarization error rate, needs care to read.
The field builds it in layers, classic to modern.
Fingerprint the voices, cluster them into speakers, then learn the whole task end to end. Newer systems handle overlap and unknown counts directly.
- 01Detect
Find speech regions.
- 02Segment
Identify candidate speaker changes.
- 03Embed
Represent each segment’s voice.
- 04Cluster
Group similar segment representations.
The classic recipe: find speech, cut at changes, fingerprint each segment, then group the fingerprints into speakers.
1Speaker embeddings
A fingerprint of a voice.
The foundation is a neural network trained to map a slice of audio to a fixed-length vector that captures who is speaking, not what they say. It is built so the same speaker lands close together and different speakers land apart. The x-vector (Snyder et al., 2018) is the widely used example: a time-delay neural network with data augmentation for robustness, which replaced the earlier i-vector approach.
This is the piece much of clustering diarization is built on. Get a good embedding and the rest of the pipeline has something solid to group.
2The modular clustering pipeline
Embed, then group.
The classic recipe runs four stages: voice activity detection, segmentation, embedding, then clustering. Clustering has historically used a PLDA similarity score with agglomerative clustering, or spectral clustering. A strong modern version, VBx (Landini et al., 2022), runs a Bayesian hidden Markov model over the x-vector sequence to find the clusters.
The honest limitation: a single hard assignment per segment means vanilla clustering struggles with overlap. When two people talk at once, one label per moment cannot represent both. Toolkits like pyannote.audio (Bredin et al., 2020) make each stage, voice activity, speaker-change detection, overlapped-speech detection, and embedding, a trainable neural component that can be optimized and recombined, which is how the modular pipeline got much stronger while staying inspectable.
| Segment | Cluster | Transcript |
|---|---|---|
| 00:00–00:03 | A | I’ll send the quote. |
| 00:03–00:05 | B | Friday works. |
| 00:05–00:08 | A | I’ll include the terms. |
3End-to-end neural diarization
One network, overlap handled directly.
An end-to-end approach reframes the whole task as one neural network that takes the mixed audio and outputs, frame by frame, the activity of each speaker as a multi-label problem: any number can be "on" at once. Trained with a permutation-invariant objective, so the arbitrary labels do not confuse scoring, it handles overlapped speech natively, which the clustering pipeline could not. The pivot is Fujita et al. (2019).
This is an end-to-end method, not a replacement for everything before it. Hybrid systems that fuse clustering with neural overlap handling remain practical and strong.
Time →
Both speakers are active between the starts and ends of their shared interval.
4Unknown and variable speaker counts
Letting the model decide how many.
Early end-to-end systems fixed the number of speakers up front. Encoder-decoder attractors, in Horiguchi et al. (2020), generate one "attractor" vector per speaker on the fly and stop when no speaker is left, so the model decides how many speakers are present instead of being told.
More recent work pushes toward streaming, or online, diarization that labels speakers as the audio arrives, and reframes the loss cleanly. The powerset multi-class formulation (Plaquet and Bredin, 2023) models overlap as a powerset of classes, which feeds into current pipelines. The throughline: the field moved from hand-built pipelines of separate parts toward single models that learn segmentation, counting, and overlap together.
- Attractor 1
Represents one inferred speaker.
- Attractor 2
Represents another inferred speaker.
- Stop decision
The model predicts that no additional speaker is needed.
5How we know it works
Diarization error rate, the field's yardstick.
The field measures itself with one headline number, diarization error rate, or DER: the share of time that is wrongly attributed. Lower is better. It sums three different mistakes: missed speech that should have been labeled, false speech labeled where there was none, and speaker confusion where the wrong speaker got the segment.
Honest limits are worth stating. DER bundles those three error types into one figure and depends on a "collar" around boundaries and on how overlap is scored, so a single number hides a lot. As an attributed point about the field, results are reported on shared benchmarks such as CALLHOME, AMI, DIHARD, and CHiME, and any DER figure belongs to the specific system and benchmark that published it, never to anyone reading this page. Compare like with like.
- Missed speech
- Reference speech was labeled as no speech.
- False alarm
- Non-speech was labeled as speech.
- Speaker confusion
- Speech was attributed to the wrong speaker.
DER = (miss + false alarm + confusion)
/ reference speaker timeSeparating the voices turns a recording into something you can use.
At the field level: a readable conversation instead of a wall of text, per-speaker analytics, search by speaker, and better downstream summaries and action items.
For a sales co-pilot, the bridge is at the outcome level. A useful record of a call is not a wall of text. It is "the buyer said this, you said that." Diarization is the quiet step that makes a transcript readable that way, and it does its work without ever needing to know anyone's name.
A readable conversation.
Transcript plus diarization equals attributed lines: you can see who said what, scan it, and quote it. See After-meeting notes.
Notes land on the right person.
Attributed turns are what let follow-ups and action items land on the right person instead of a blur, so what the rep committed to and what the client asked for stay separate.
Search and recall by speaker.
"Find where the customer raised the objection" only works once the customer's turns are separable from everyone else's.
Multi-party calls become tractable.
Group calls and room recordings turn into a structured record instead of an undifferentiated blur, the hard case from above made usable.
The direction the field is heading.
We describe the direction, not a roadmap of our own, and stay honest about the open problems.
One model, or both fused.
From modular clustering pipelines toward end-to-end models, and toward hybrids that fuse clustering with neural overlap handling to combine the strengths of each.
Labeling speakers live.
From offline batch processing toward streaming, or online, diarization that labels speakers as the audio arrives, with low latency.
Audio plus video.
Multimodal diarization that adds lip motion and who is on camera for in-room and meeting settings, where the picture helps settle the audio.
Who and what, together.
Tighter joint modeling of recognition and diarization, so "who spoke" and "what they said" are solved together rather than stitched after the fact.
The honest hard cases.
Dense overlap, very short turns, large and unknown speaker counts, robustness across far-field rooms and telephone audio and many languages, and an evaluation metric that compresses several failures into one number.
The papers behind this page.
- Snyder, Garcia-Romero, Sell, Povey, Khudanpur (2018). X-Vectors: Robust DNN Embeddings for Speaker Recognition. ICASSP 2018, DOI 10.1109/ICASSP.2018.8461375.
- Landini, Profant, Díez, Burget (2022). Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization. arXiv:2012.14952.
- Bredin, Yin, Coria, et al. (2020). pyannote.audio: neural building blocks for speaker diarization. arXiv:1911.01255.
- Fujita, Kanda, Horiguchi, Nagamatsu, Watanabe (2019). End-to-End Neural Speaker Diarization with Permutation-Free Objectives. arXiv:1909.05952.
- Horiguchi, Fujita, Watanabe, Xue, Nagamatsu (2020). End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors (EEND-EDA). arXiv:2005.09921.
- Plaquet, Bredin (2023). Powerset multi-class cross entropy loss for neural speaker diarization. arXiv:2310.13025.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
Separating the voices is the quiet step that turns a recording into something you can read, search, and act on. See it on a real call.
No app required. Just your phone number.