Self-supervised and contrastive learning.
A model can learn the structure of language, speech, and images from raw data, before any person labels a thing. Here is how that works, in plain language.
Most machine learning needs a human answer key: thousands of examples, each labeled by hand. That key is expensive, and for a lot of real data it does not exist. Self-supervised learning drops the key and lets the model build its own quiz out of the data. Contrastive learning is one popular way to set that quiz up. We walk it the way the field is built: why labels are the bottleneck, how a model writes its own quiz, how "pull together, push apart" shapes a useful space, what pre-train then fine-tune makes possible, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
The model makes its own quiz from the data.
Supervised learning needs a human answer key. Self-supervised learning builds the training signal out of the raw data itself.
Picture machine learning as a teacher with an answer key. You show the model thousands of examples, each one labeled by a person ("this is a cat," "this call was a win," "this field is the renewal date"), and it learns to copy the answers. That works, but the key is expensive, and for a lot of real data there is simply no key.
Self-supervised learning removes the teacher. Instead of asking a person for labels, the model invents a task whose answer is already present in the data. Hide a word in a sentence and ask it to guess the word. Cut out a stretch of audio and ask it to fill it back in. The answer was sitting in the data the whole time, so no human had to write it down.
The point is not the guessing game. It is what the model has to learn in order to play it well: which words go together, how a sentence tends to turn, what a voice sounds like mid-thought. That learned understanding, a representation, is the prize, and it is what gets reused. The pattern that frames the whole field is two stages: pre-train on a large pile of unlabeled data with a self-made task, then fine-tune on the small labeled dataset for the job you actually care about.
A running example for this page: a heap of call transcripts and notes with almost no labels, and the question of how a model could learn anything useful from it before any person labels a single record.
The meeting starts at noon.
The meeting starts at MASK.
Two things make it hard: the supply of labels, and the risk of cheating.
Labels are scarce, and a quiz the model can shortcut teaches nothing. Both shape every method below.
Labels are the scarce thing.
Hand-labeling is slow and costly, and for most real data, raw audio, free-text notes, conversation, no label exists at all. The supply of unlabeled data dwarfs the supply of labeled data.
An easy task teaches a shortcut.
A pretext task that is too easy lets the model cheat without learning structure. Designing a task that forces genuine understanding, not a trick, is the hard part.
Map everything to one point.
If you only reward the model for calling matching things similar, it can cheat by sending everything to the same place. Everything is then trivially similar to everything. Stopping this collapse is a central engineering problem.
"Different" examples cost something.
Contrastive methods need negatives to push away, and naively they need a lot of them, which means large batches or extra bookkeeping. Where the negatives come from is its own design question.
More from less, on real data.
Most of a company's conversation data arrives unlabeled. As an example of the idea, not a claim about any product: a method that learns the shape of the material first, then adapts on the small labeled set a team can realistically produce, is how the field gets more from less.
Hide part of the data. Predict it back. Pull together, push apart.
A few recipes carry the field: predict the hidden part, contrast matching pairs against the rest, then pre-train once and fine-tune for the job.
1The pretext task: predict the hidden part
Hide a word, guess the word.
The model is shown an input with a piece removed and asked to reconstruct it. In text, the landmark version is masked language modeling: BERT (Devlin et al., 2018) randomly hides a fraction of the words in a sentence and trains the model to predict them from both the left and the right context at once. Predicting the masked word well requires understanding grammar, meaning, and context, which is exactly the representation you want.
The same idea works in audio. wav2vec 2.0 (Baevski et al., 2020) masks spans of the speech signal in a latent space and trains the model to identify the correct content for the masked spans against distractors. It learns speech representations from audio alone, then needs only a small amount of transcribed speech to fine-tune. A clean example of pre-train on raw, fine-tune on little.
Corrupted input
Target from the original text
2Contrastive learning
Pull together, push apart.
Make two views of the same item. For an image, two crops or color shifts, called augmentations. These two form a positive pair and should land close in the representation space. Other items in the batch are negatives and should be pushed away. SimCLR (Chen et al., 2020) showed that this simple recipe, with the right augmentations and a large enough batch, learns image representations strong enough that a plain linear classifier on top approaches supervised performance, a result reported in that paper.
The push-apart force is what stops collapse. Without negatives to repel, the model could map everything to one point. So "different" examples are not a detail. They are what makes the space useful.
Schematic representation space.
3Where the negatives come from
A running dictionary of "different."
SimCLR got its negatives from large batches, which is costly. MoCo (He et al., 2019) reframes contrastive learning as looking things up in a dictionary, and builds that dictionary as a running queue of recent examples encoded by a slowly moving, or momentum, copy of the model. This gives a large, consistent supply of negatives without enormous batches.
Then comes the turn that surprised the field. BYOL (Grill et al., 2020) showed you can learn strong representations with no negative pairs at all, using two networks: an online network that tries to predict the output of a slow-moving target network on another view of the same image. The asymmetry plus the moving target keeps the model from collapsing, and opened a whole family of positive-only methods.
SimCLR
- Source
- Other examples in a large training batch.
- Trade-off
- Batch size affects the set of comparisons.
MoCo
- Source
- A queue of recent encoded examples.
- Stability
- A momentum encoder changes more slowly.
4Pre-train, then fine-tune
Learn structure once. Spend the labels last.
Putting it together: pre-train on a large unlabeled corpus with a self-supervised task to get a general representation. Then fine-tune on the small labeled dataset for the target job. The heavy lifting happens once, on cheap data. The scarce labeled data is spent only on the final adaptation.
As an example of the idea, not a claim about any product: you could take a broadly capable model and adapt it toward one organization's vocabulary and way of selling using that organization's own unlabeled conversations, rather than waiting for a hand-labeled dataset that may never exist. The same two-stage shape is also a natural way to keep one company's material the basis for its own adaptation, separate from another's. These are illustrations of the pattern, not statements about how anything we build is made.
- Unlabeled corpus
Create prediction targets from the data itself.
- Pretrained representation
Learn reusable structure.
- Task-specific data
Use labeled examples or another adaptation method.
- Evaluation
Measure the intended downstream behavior.
What it makes possible: learning from data nobody labeled.
At the field level: useful starting models from raw text, audio, and images, more from less labeled data, and reusable backbones that many tasks fine-tune from.
This family of techniques is how the industry teaches models from raw, unlabeled material before anyone hand-labels it. The lines below are the kind of outcome that makes possible for a seller, stated as outcomes and as examples of the idea, never as a claim about how any system we build is put together.
Learn from the material you already have.
A model can learn the shape of sales conversation from the raw calls, notes, and transcripts a team already holds, instead of waiting on a labeling project that may never finish.
More from less labeled data.
After pre-training, a task can be reached with a fraction of the labels it would otherwise need. wav2vec 2.0 reported strong speech recognition from very little transcribed audio after pre-training on a large unlabeled set.
Adapt a general model to one domain.
The two-stage pattern is a natural way to tune a broadly capable model toward one field's vocabulary and conventions. See the sibling contrastive-training beat in Embeddings and vector search.
One idea across formats.
"Learn structure from the data itself" applies to text, speech, and images alike, which is part of why these methods underpin so much of modern model building.
This explainer covers published research. To see how AI REV Voice helps with sales conversations, explore Prism.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
The same idea, everywhere.
Masked-prediction and contrastive ideas now span text, speech, vision, and combinations of them, pushing toward shared approaches rather than one trick per data type.
Less hand-tuning of augmentations and negatives.
Non-contrastive and masked-reconstruction methods reduce the need for carefully chosen augmentations or large negative sets, simplifying the recipe.
Self-supervision as standard pre-training.
The pre-train then fine-tune pattern that self-supervision established is now the common backbone of large models.
Theory, not folklore.
Active study of collapse, what augmentations actually teach, and what makes a representation transfer, so the recipes rest on understanding rather than habit.
Strong representations for less.
Getting useful representations with less compute and less data, so the approach reaches more domains and smaller settings.
The papers behind this page.
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805.
- Baevski et al. (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. arXiv:2006.11477.
- Chen et al. (2020). A Simple Framework for Contrastive Learning of Visual Representations (SimCLR). arXiv:2002.05709.
- He et al. (2019). Momentum Contrast for Unsupervised Visual Representation Learning (MoCo). arXiv:1911.05722.
- Grill et al. (2020). Bootstrap your own latent: A new approach to self-supervised Learning (BYOL). arXiv:2006.07733.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
You talk, and clean, complete notes are waiting. Hear a co-pilot answer out loud on a real call.
No app required. Just your phone number.