Knowledge Distillation
Pouring a big model's judgment into a smaller one
- Knowledge distillation makes a small model imitate the judgment of a strong, large model, so a small body ends up with similar performance.
- The small side doesn't just learn from the right answer. It also learns from how split the large model's confidence was among the candidates.
- That split carries information no answer key ever holds — which options get confused with each other, and which never do.
- When the split looks too lopsided to reveal anything, the scale gets stretched out to expose the differences before the small model learns from it.
- It's used alongside other ways of shrinking a model, aimed at the same goal as coarsening the scale numbers are written on, or cutting away connections that are rarely used.
Contents
1The analogy
A large sorting machine out at the farm weighs, measures sweetness, checks color, and scans for blemishes on every single piece of fruit that passes through, then grades it. A neighborhood store can't install that machine — there's no room, and no budget. So here's what happens instead: thousands of pieces of fruit get run through the big machine, and its full verdict gets written down, not just the final grade. Then a small machine with nothing but a camera gets shown that record until it starts landing on the same verdicts. There's one trick here — the big machine's hedged calls, like "mostly Premium, but some chance of Standard too," get handed over exactly as they were. That hesitation is where the distance between grades lives. Knowledge distillation is this act of pouring that judgment from one machine into another.
2In detail
The large one teaches, the small one learns
Distillation always involves two models: an already well-trained large one, and a new small one being built. The large one is called the teacher, the small one the student. The same training data goes into both, and only the student gets adjusted, nudged closer to what the teacher outputs. The teacher is never touched.
A major advantage here is that the data doesn't need answers attached at all. Any data run through the teacher produces something to learn from, even if no person ever tagged it. Answered data is always in short supply; unanswered data is everywhere.
What the answer key never held
Learning from the answer alone passes along a single fact: "this piece is Premium." Every other grade gets treated as equally wrong. A teacher's verdict carries far more than that — leaning heavily toward Premium, still carrying some weight toward Standard, with almost nothing left for Reject.
That shape reveals the distance between grades. Premium and Standard are plausibly confusable; Premium and Reject never are — and that fact gets etched into the shape of the numbers. The student learns to imitate that whole shape rather than a single answer, and ends up picking up similar judgment from far less data.
Stretching the scale out to make it visible
A well-trained teacher tends to lean hard toward one side. Almost everything piles onto Premium, leaving only tiny scraps for the rest. Hand that over as-is, and it isn't much different from showing the student a single answer — whatever information was packed in gets buried under how small the numbers are.
So the scale gets stretched out for a moment. The gaps between values get smoothed out so the small values become visible, and only then does the student see it. The student's own output gets measured on that same stretched scale, and everything snaps back to the normal scale once training wraps up. Stretch too far and every grade looks about the same, leaving nothing to learn; keep it too narrow and the information stays buried. Tuning that width is the trick to distillation.
How much can you shrink it
You can shrink a model a great deal while holding onto most of its performance. Students get built with fewer layers, or narrower ones, and sometimes the verdicts of several teachers get averaged together before handing them over. Matching the shape of the middle layers, not just the final output, often transfers even better.
What you gain is speed and cost. A small model answers faster, uses less memory, and can run on a laptop or a handheld device. It's often used alongside other ways of shrinking a model — coarsening the scale numbers are written on to shrink storage, and cutting away connections that are rarely used — which aim at the same goal without overlapping with distillation.
What doesn't carry over
What distillation carries over is judgment already inside the range the teacher handles well. Wherever the teacher gets confused, the student gets confused the same way. Whatever bias or bad habit the teacher has comes along too. Since the student's whole goal is imitating the teacher, if the teacher leans wrong, the student learns that lean right along with it.
The gap can reopen on tasks that demand broad, deep knowledge. In common situations, the small model lands close to the teacher's answer, but the size difference shows up in rare, tricky ones. So distillation isn't really a way to replace the large model — it's closer to building a cheap partner that handles the common cases.
3More precisely
Knowledge distillation trains a student model to follow a teacher model's output distribution. An answer that marks only one correct option is called a hard label; the distribution the teacher produces is called a soft label. The value that smooths the distribution out is called temperature, and the student's loss is often a blend of a term that follows the soft label and a term that gets the actual answer right. Matching intermediate representations rather than just the output is called feature distillation.
The analogy breaks down in places. A fruit sorter measures fixed, well-defined things like weight and sweetness, but nobody can write down as a checklist what a large model actually bases its judgment on. So what gets poured over is the shape of the result, not the reasoning behind it — the student never learns why, only what. There's another gap: a sorting machine has a scale as its solid ground truth, but a teacher model's verdict is itself just a guess, with no fruit and no scale standing behind it. Wherever the teacher is wrong, the student earnestly learns the wrong answer too, and there's no independent check inside the process to catch that drift.
4Try it yourself
5Common misconceptions
It's easy to think distillation makes the small model as good as the large one, but actually it only gets close on common situations — the gap still shows up on rare, difficult ones.
It's easy to think distillation is a technique for compressing a model, but actually it isn't shrinking the large model at all — it's training an entirely new, small model.
It's easy to think it's enough to learn from the teacher's answer alone, but actually the real benefit of distillation only shows up once the split among the candidates gets passed along too.
7One-line summary
In shortKnowledge distillation trains a small model to imitate the shape of a large model's judgment, so a small body ends up giving a similar answer.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02