Key points
- Knowledge distillation makes a small model imitate the judgment of a strong, large model, so a small body ends up with similar performance.
- In the basic method, the small side doesn't just learn from the right answer. It also learns from how split the large model's confidence was among the candidates.
- That split carries information no answer key ever holds — which options get confused with each other, and which never do.
- When the split looks too lopsided to reveal anything, the scale gets stretched out to expose the differences before the small model learns from it.
- It's used alongside other ways of shrinking a model, aimed at the same goal as coarsening the scale numbers are written on, or cutting away connections that are rarely used.
The analogy
A large sorting machine out at the farm weighs, measures sweetness, checks color, and scans for blemishes on every single piece of fruit that passes through, then grades it. A neighborhood store can't install that machine — there's no room, and no budget. So here's what happens instead: tens of thousands of pieces of fruit get run through the big machine, and its full verdict gets written down, not just the final grade. Then a small machine that looks at the same fruit but has far fewer parts gets shown that record until it starts landing on the same verdicts. There's one trick here — the big machine's hedged calls, like "mostly Premium, but some chance of Standard too," get handed over exactly as they were. That hesitation is where the distance between grades lives. Knowledge distillation is this act of pouring that judgment from one machine into another.
In detail
The large one teaches, the small one learns
Distillation always involves two models: an already well-trained large one, and a new small one being built. The large one is called the teacher, the small one the student. The same training data goes into both, and only the student gets adjusted, nudged closer to what the teacher outputs. The teacher is never touched.
A major advantage here is that the data doesn't need answers attached at all. Any data run through the teacher produces something to learn from, even if no person ever tagged it. Answered data is always in short supply; unanswered data is everywhere.
What the answer key never held
Learning from the answer alone passes along a single fact: "this piece is Premium." Every other grade gets treated as equally wrong. A teacher's verdict carries far more than that — leaning heavily toward Premium, still carrying some weight toward Standard, with almost nothing left for Reject.
That shape reveals the distance between grades. Premium and Standard are plausibly confusable; Premium and Reject never are — and that fact gets etched into the shape of the numbers. The student learns to imitate that whole shape rather than a single answer, and ends up picking up similar judgment from far less data.
Stretching the scale out to make it visible
A well-trained teacher tends to lean hard toward one side. Almost everything piles onto Premium, leaving only tiny scraps for the rest. Hand that over as-is, and it isn't much different from showing the student a single answer — whatever information was packed in gets buried under how small the numbers are.
So the scale gets stretched out for a moment. The gaps between values get smoothed out so the small values become visible, and only then does the student see it. The student's own output gets measured on that same stretched scale, and everything snaps back to the normal scale once training wraps up. Stretch too far and the student ends up copying the noise in the teacher's tiny, loosely assigned values; keep it too narrow and the information stays buried. Tuning that width is the trick to distillation.
How much can you shrink it
You can shrink a model a great deal while holding onto most of its performance. Students get built with fewer layers, or narrower ones, and sometimes the verdicts of several teachers get averaged together before handing them over. Matching the shape of the middle layers, not just the final output, often transfers even better.
What you gain is speed and cost. A small model answers faster, uses less memory, and can run on a laptop or a handheld device. It's often used alongside other ways of shrinking a model — coarsening the scale numbers are written on to shrink storage, and cutting away connections that are rarely used — which aim at the same goal without overlapping with distillation.
What doesn't carry over
What distillation carries over is judgment already inside the range the teacher handles well. Wherever the teacher gets confused, the student tends to get confused in much the same way. Whatever bias or bad habit the teacher has comes along too. Since the student's whole goal is imitating the teacher, if the teacher leans wrong, the student learns that lean right along with it.
The gap can reopen on tasks that demand broad, deep knowledge. In common situations, the small model lands close to the teacher's answer, but the size difference shows up in rare, tricky ones. So distillation isn't really a way to carry over the large model's full ability — it's closer to building a separate small model that handles the common cases cheaply. When the hard cases have to be covered too, the large model is sometimes kept alongside.
More precisely
Knowledge distillation trains a student model to follow a teacher model's output distribution. An answer that marks only one correct option is called a hard label; the distribution the teacher produces is called a soft label. The value that smooths the distribution out is called temperature, and the student's loss is often a blend of a term that follows the soft label and a term that gets the actual answer right. Matching intermediate representations rather than just the output is called feature distillation. In today's language models, using the answer text a teacher wrote directly as training material to fine-tune the student is also called distillation, and is widely used.
The analogy breaks down in places. A fruit sorter measures fixed, well-defined things like weight and sweetness, but nobody can write down as a checklist what a large model actually bases its judgment on. So what gets poured over is the shape of the result, not the reasoning behind it — the student never learns why, only what. There's another gap: a sorting machine has a scale as its solid ground truth, but a teacher model's verdict is itself just a guess, with no fruit and no scale standing behind it. Wherever the teacher is wrong, the student is prone to learning the wrong answer too, which is why a term graded against the real answer is often mixed in.
Try it yourself
Common misconceptions
It's easy to think distillation makes the small model as good as the large one, but actually it only gets close on common situations — the gap still shows up on rare, difficult ones.
It's easy to think distilling makes the large model itself smaller, but actually the large model stays as it is, and distillation is a form of compression done by training an entirely new, small model.
It's easy to think learning from the teacher's answer alone is all there is to distillation, but actually passing along the split among the candidates carries over more information than the answer key holds. Learning from the answers the teacher produces alone also works well, though, and is widely used.
One-line summary
Knowledge distillation trains a small model to imitate the shape of a large model's judgment, so a small body ends up giving a similar answer.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-17