Reading sentiment and intent in a conversation.
A single line rarely tells you how a conversation is going. "Sure, send it over" can be an eager yes or a polite stall. The meaning lives in what came before it, in who is speaking, and in how it was said.
This is a plain-language guide to two threads that run through any dialogue: sentiment, how positive or negative the mood is and whether it is warming or cooling, and intent, what a speaker is actually trying to do. The field that studies them across the turns of a conversation is called Emotion Recognition in Conversation, or ERC. We walk it the way the field is built: what it is, why context makes it hard, how modern systems model a dialogue and its speakers, what that unlocks, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
Two threads, read across the turns.
The everyday job is to track mood and goal as a dialogue unfolds. The load-bearing phrase is "in a conversation."
Sentiment is how positive or negative a turn is, often on a scale rather than a hard label, and crucially its trajectory over the conversation. Intent is the communicative goal of a turn: to ask, agree, object, request, or defer. Emotion is a richer layer on top, the joy, anger, frustration, or surprise that the ERC literature usually labels turn by turn.
Classic sentiment analysis labels one sentence or one document on its own. ERC re-frames the unit of analysis as the dialogue: each turn is read in light of the turns around it. The same words carry different meaning depending on what preceded them, so "Sure, send it over" can be a green light or a brush-off. Three things change meaning beyond the words themselves: the conversational context, the speaker dynamics, and, when audio is present, the tone. The single-voice side of tone, pitch and rhythm and emotion from audio, has its own page in Speech emotion. This page is about the thread that runs between utterances.
“That solves the problem we described.”
“Sure, send it over.”Likely interest
“We’re still comparing a few options.”
“Sure, send it over.”Interest is less certain
Meaning lives in context, not in isolated lines.
A conversation is not a stack of independent sentences. It carries history, speakers, and tone, and any of them can flip what a turn means.
The cue may be turns back.
The history that settles a turn might be the line right before it, or it might be several turns earlier. Models have to learn which part of the history matters, not just average it all together.
Mood has inertia and contagion.
A speaker carries their own running state across their turns, and the other party nudges it. Treating each turn as independent throws away how feeling persists and spreads between people.
Annotations disagree and blur.
People label the same turn differently, and a catch-all category like "neutral" absorbs a lot. The ERC datasets document real class imbalance and ambiguity, so the ground truth is itself soft.
"Great." can mean its opposite.
Irony inverts the words on the page, and short backchannels like "mm-hm" or "right" carry intent while holding almost no content. Both lean hard on context to be read correctly.
The words and the tone diverge.
The transcript says one thing and the voice says another. Fusing the two well, so neither drowns out the other, is its own research problem rather than a simple concatenation.
A live read has to keep up.
Reading a conversation as it happens, rather than after the fact, is named in the survey as an explicit challenge. On a live call the value of a read decays fast, so the work has to keep pace with the talk.
Model the dialogue, the speakers, and the tone.
The field's progress comes from a few clear shifts: better data, per-speaker state, context as a graph, and fusing what was said with how it was said.
1The data made the field possible
Conversations, labeled turn by turn.
Progress needed corpora that look like real dialogue. MELD (Poria et al., 2019) extended a set of multi-party conversations into a multimodal corpus of roughly thirteen thousand utterances, each turn carrying text, audio, and video, and labeled with emotion and sentiment. Multi-party, meaning more than two speakers, is what forced models to reason about who is speaking.
For the single-speaker setting, CMU-MOSEI (Zadeh et al., 2018) supplied large-scale multimodal sentiment and emotion-intensity labels, roughly twenty-three thousand annotated segments from a thousand speakers, the scale that let later models generalize.
Fine. Whatever works.
- Previous turn
- A request to change an already agreed deadline.
- Possible signals
- Flat delivery, short response, reluctant wording.
Treat the label as a contextual estimate, not a fact about the person.
2Model the speaker, not just the sequence
Follow each person through the talk.
DialogueRNN (Majumder et al., 2019) tracks a separate party state for each speaker alongside a global context, updating them turn by turn so the model adapts to who is talking. That was the shift from "label each utterance on its own" to "follow each participant through the dialogue," which captures how a mood persists for one person and is nudged by the other.
Knowing reliably which lane a turn belongs to is its own task, who spoke when, handled in Speaker diarization. ERC builds on that to track how each speaker is feeling over time.
| Turn | Utterance | State evidence |
|---|---|---|
| A1 | We can begin next week. | Positive expectation |
| B1 | We are weighing other options. | Reservation |
| A2 | Of course. What is still open? | Acknowledges reservation |
3Model the conversation as a graph
Context is a web of who said what.
DialogueGCN (Ghosal et al., 2019) represents the dialogue as a graph whose edges encode two kinds of dependency: self-dependency, a speaker's own earlier turns, and inter-speaker dependency, how participants influence each other. Letting context propagate along those edges reaches relevant turns that sit past the distance limits of a plain sequence model.
The practical payoff is a read that can connect a turn to the objection three turns back, or to the speaker's own earlier promise, rather than only to the line immediately before it.
A1 → B1 → A2 → B2
A1 ─────→ A2 (same speaker)
B1 ─────→ B2 (same speaker)- Sequential edges
- Carry the order of the exchange.
- Speaker edges
- Connect what one person said across multiple turns.
4Fuse what was said with how it was said
The transcript is half the message.
When tone is available it disambiguates the words, so systems combine the modalities. Tensor Fusion Networks (Zadeh et al., 2017) model both the intra-modality dynamics, within text and within audio, and the inter-modality dynamics, how the two interact, rather than just concatenating features. CMU-MOSEI introduced the interpretable Dynamic Fusion Graph in the same spirit.
Intent and dialogue-act recognition carry the same lesson: context and speaker matter. We keep this page anchored on the sentiment and emotion literature and treat intent as the parallel, goal-oriented reading rather than claiming one unified model for both.
Text
Sure, that’s fine.
Audio evidence
- Delivery
- Clipped words, a pause, falling pitch.
Consider agreement or reluctance, then clarify if the distinction matters.
A running read, not a single verdict.
Tracking sentiment across turns shows direction, and reading more than one speaker captures how a real conversation actually moves.
| Moment | Possible contextual signal |
|---|---|
| Early discussion | More engaged responses |
| Objection | More hesitation |
| After clarification | Renewed engagement |
As a field-level example, picture a live sales call. As the two people talk, a co-pilot could track the running sentiment, is the buyer warming up or pulling back, and flag intent as it surfaces: a pricing question, a competitor mention, a soft yes. Because a conversation moves second by second, the value is in reading it as it happens and getting that fresh read to wherever the rep is working quickly, so the next thing they say lands. The same idea scales to a fast-updating app that rolls up many calls. All of this is framed as what the field enables, never as a claim about any product.
Direction, not just a label.
Tracking sentiment across turns shows warming, cooling, or recovery after an objection, which a one-shot label cannot capture.
Reading the room.
Multi-party modeling supports realistic conversations where several participants react to each other, not just a single voice in isolation.
Tone-aware understanding.
Fusing audio with text catches what the transcript misses: the sarcastic "great," the hesitant "sure." The single-voice signal lives in Speech emotion.
Useful while it still matters.
A live read only helps if it arrives in time. Knowing when a turn has ended, so a read can be acted on, is covered in Turn detection.
This explainer covers published research. To see how AI REV Voice helps with sales conversations, explore Prism.
The direction the field is heading.
We describe the direction the research is taking, drawn from the survey, not a roadmap of our own.
From RNNs to graphs to large models.
Context modeling keeps getting richer, moving from per-speaker recurrent states to graphs to transformer- and language-model-based context encoders, with the survey mapping the open problems.
Reading as it unfolds.
Streaming ERC, reading a conversation as it happens rather than after the fact, is named in the survey as an explicit challenge the field is still working on.
Why a feeling occurred.
Beyond labeling emotion, the field is moving toward the reason behind it, emotion-cause analysis, and toward handling sarcasm and figurative language more reliably.
Consent, transparency, and fairness.
Inferring mood and intent from a conversation is sensitive. The field is developing norms on consent, transparency, and appropriate use, and working on consistency across speakers, accents, and cultures.
The papers behind this page.
- Majumder et al. (2019). DialogueRNN: An Attentive RNN for Emotion Detection in Conversations. arXiv:1811.00405.
- Poria et al. (2019). MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. ACL 2019.
- Ghosal et al. (2019). DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation. EMNLP-IJCNLP 2019.
- Zadeh et al. (2017). Tensor Fusion Network for Multimodal Sentiment Analysis. EMNLP 2017.
- Zadeh et al. (2018). Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. ACL 2018.
- Poria et al. (2019). Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances. IEEE Access. arXiv:1905.02947.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
A conversation moves second by second. See what a real call sounds like when the read keeps up.
No app required. Just your phone number.