Models & training
Knowledge distillation.
How a small, fast model is trained to behave like a large, capable one. A teacher teaches a student not just the answer, but how confident to be and why.
Knowledge distillation is a way to compress what a big model knows into a smaller one. The large model is the teacher. The small one is the student. Instead of copying only the final answers, the student learns from the full shape of the teacher's confidence. We walk the field the way it is built: what it is, why it matters, how it works, what it unlocks, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
Teaching a small model to behave like a big one.
The plain idea: train a small student to reproduce the behavior of a large teacher, so you can deploy the small one.
Picture a senior account executive who has closed a thousand deals. Ask why a prospect is unlikely to buy and the answer is not a flat yes or no. It is a read: probably not this quarter, maybe next, watch the budget signal. A new hire who only ever sees the final outcome of past deals, won or lost, learns less than one who hears the full reasoning, including how unsure the senior rep was and which near-misses looked similar. The gradations carry information the bare outcome throws away.
Distillation is that idea for models. A trained model's output is more than its top answer: the full probability it spreads across every option carries information about how it generalizes, for example that a given "7" looks a little like a "1." Hinton, Vinyals, and Dean (2015) call this the dark knowledge. The student is trained on these soft targets, usually alongside the real hard labels. Soft targets carry more information per example, so a much smaller model can reach close to the teacher's quality while running faster and cheaper.
The model that wins on quality is often too big to ship.
Training and serving are different jobs with different constraints. Distillation exists to close that gap, and several things make it hard.
Big models are expensive to run.
The model that wins on quality is often too large or too slow for where it has to live: a phone, a browser, a real-time voice loop, a high-volume server. Many large models behave like a larval form tuned for one job and an adult form tuned for another.
A small model trained alone loses quality.
Training a small model from scratch on the same labels usually underperforms, because hard labels alone are a thin signal. The hard part is transferring the generalization the big model learned, not just its final answers.
Temperature is a trade-off.
Soft targets come from raising the softmax temperature so the probabilities spread out and the small wrong-class numbers become visible. Set it well and the student sees rich structure. Set it badly and you wash out signal or amplify noise.
Teacher and student have to fit.
A student far smaller than its teacher may lack the capacity to match it. A weak or miscalibrated teacher passes on its mistakes. Choosing a teacher and a student that suit each other is its own design problem.
How the knowledge gets transferred.
Soften the teacher's output, train the student to match it, and choose what to copy: final answers, inner features, or the relations between examples.
1Response-based distillation
Match the teacher's confidence, not just its decision.
The original method trains the student to match the teacher's output probabilities under a raised temperature, using a distillation loss, typically KL divergence to the soft targets, combined with the ordinary loss on the true labels. The student learns the teacher's full confidence structure, not only which class it picked. This is the spine of the field, from Hinton, Vinyals, and Dean (2015).
The contrast is easy to see. A hard label says one class is right and every other is wrong. A soft target says a given email looks 70% like a renewal, 25% like an upsell, and 5% like churn. That spread tells the student which things look alike and how sure to be.
Hard target
Soft target
2The temperature knob
Turn the dial until the structure shows.
Temperature controls how much of the teacher's nuance the student can see. At low temperature the distribution is a single spike, almost the same as a hard label. Raise it and the small bars lift into view, revealing which wrong answers the teacher considered plausible. Raise it too far and everything flattens toward noise. The art is setting it so structure shows without washing out, which is why the original work treats it as the central knob.
| Class | Lower T | Higher T |
|---|---|---|
| Renewal | 88% | 48% |
| Expansion | 10% | 32% |
| Churn | 2% | 20% |
Illustrative normalized distributions, not measured outputs.
3Where the knowledge comes from
Copy the answers, the inner features, or the relations.
Response-based distillation copies the final output. Two other taps exist. Feature-based distillation has the student also match the teacher's intermediate representations, the hints from a middle layer, which guides a thin, deep student through training that would otherwise be hard to optimize, as in FitNets (Romero et al., 2015). Relation-based distillation transfers the relationships the teacher sees between examples, which inputs it treats as similar, rather than per-example outputs. The 2021 survey (Gou et al.) organizes the field around these three knowledge types.
- Responses
- Match the teacher’s output distribution.
- Features
- Match intermediate representations.
- Relations
- Match similarities or relationships across examples.
4Training schemes
When the teaching happens.
Beyond what is transferred, there is when. In offline distillation the teacher is fixed and pre-trained, and the student learns from it. In online distillation the teacher and student train together. In self-distillation a model teaches a smaller or later version of itself. The 2021 survey (Gou et al.) is the reference for how these fit together.
| Scheme | Training arrangement |
|---|---|
| Offline | A previously trained teacher supervises a student. |
| Online | Models learn with each other during training. |
| Self-distillation | A model’s outputs or representations supervise another stage or part of itself. |
5A worked language-model case
What this looks like when it ships.
DistilBERT (Sanh et al., 2019) applies response-based distillation during pre-training with a triple loss: language modeling, distillation, and a cosine-distance term that aligns hidden states. The paper reports a model about 40% smaller and 60% faster that keeps roughly 97% of the teacher's benchmark score. Those figures are that paper's reported result, framed as one concrete case, not a generic guarantee. They are the language model's numbers, not anyone else's.
A small model with a big model's judgment.
At the field level, distillation is a standard route to a model that is deployable, not just accurate on paper: most of the quality, at a fraction of the size and run cost.
The points below are examples of where this idea appears in the world, not claims about how any AI REV system is built. They show why responsive, affordable assistance at scale is feasible at all.
Models that fit where they live.
Tight latency budgets, constrained hardware, and high request volumes are exactly the constraints a distilled student is built for. The field uses distillation to get a usable model, not only an accurate one.
Cheaper, faster serving.
A smaller student means less memory and compute per request, which compounds at scale. This is where the training-time idea hands off to serving-time work in Low-latency inference.
Specialization.
A general teacher's knowledge can be distilled into a focused student. As an illustrative example: adapting a capable general model toward one organization's way of selling, or toward a narrow task, while keeping the footprint small. Example, not a product claim.
Confidence you can keep honest.
Because the method transfers confidence, keeping the student's probabilities well-calibrated, so a 70% means 70%, is part of doing it well. Useful wherever a model's certainty is read, not just its answer.
This explainer covers published research. To see how AI REV Voice helps with sales conversations, explore Prism.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Strong models as teachers.
Using a capable model to generate training signal, soft labels or full reasoning traces, for a smaller one is now a mainstream way to build compact, capable models. A current direction, grounded in the survey and the field's broader work.
More feature and relation transfer.
Continued growth of feature-based and relation-based transfer, and of self-distillation, as ways to close the gap between a small student and a much larger teacher.
Distill, then quantize.
Distillation composes with other compression. A common pattern is to distill a smaller student and then quantize it to hit a size and latency target. The serving-side details live in Low-latency inference.
Keeping confidence meaningful.
Because the method transfers confidence, interest is growing in keeping the student's probabilities well-calibrated, so a stated certainty can be read at face value.
The papers behind this page.
- Hinton, Vinyals, Dean (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531.
- Romero, Ballas, Kahou, Chassang, Gatta, Bengio (2015). FitNets: Hints for Thin Deep Nets. arXiv:1412.6550.
- Sanh, Debut, Chaumond, Wolf (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108.
- Gou, Yu, Maybank, Tao (2021). Knowledge Distillation: A Survey. arXiv:2006.05525.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
Smaller, faster, sharper models are part of why a co-pilot can keep up with a real conversation. See it on a real call.
No app required. Just your phone number.