Key points
- Pruning removes connections inside a model that barely contribute to the result. It separates what stays from what goes.
- A trained model ends up with an enormous number of connections whose value sits near zero. Cutting them usually barely changes the answer, but some small ones sit in important spots, so the way you pick what to cut matters.
- Nobody cuts a huge chunk all at once. Cut a little, then retrain, over and over — that's what keeps performance from dropping too much. In today's large language models, cutting in one shot without retraining while correcting the remaining values is also widely used.
- There's cutting scattered connections here and there, and there's removing whole rows or blocks at once. It's usually the wholesale kind that actually speeds things up.
- It's a different move from quantization, which coarsens the scale numbers are written on. Pruning reduces the count itself.
The analogy
A food stall's menu board lists about eighty items. But tally up a year's worth of orders and half of them turn out to have never been ordered even once — names still hanging on the board while the ingredients quietly go bad in the fridge.
So the never-ordered items get struck from the board. Ingredient space opens up, the kitchen workflow gets shorter, and there's less fumbling when an order comes in. What customers actually get barely changes — they were only ever ordering what they always ordered anyway.
Cut forty items at once, though, and trouble follows. Even the odd dish someone orders once in a while disappears, and regulars start walking away. Safer to cut about ten, watch for a month, then cut ten more. Pruning is this same process of trimming away what isn't used.
In detail
Why so many unused connections pile up
A model isn't built to be exactly the right size from the start. It's built generously oversized on purpose and then trained — going in big tends to make training go better. So once training wraps up, what's left is a mix: connections that genuinely carry weight toward the answer, and connections whose value just hovers near zero.
A connection near zero contributes almost nothing to the result no matter what it's multiplied against. And yet it still takes up its spot in the calculation and still triggers a multiplication every single time. It's not unusual for connections like this to make up more than half the total.
Pruning picks out these connections and cuts them. What's left behind at each cut spot is a zero. Tidy those zeros up separately or use a dedicated way of computing, and they can be skipped; left as they are, the amount of computation stays the same.
How to pick what to cut
The most common yardstick is simply the size of the value. Line everything up from closest to zero and cut from the bottom. It takes little effort and works reasonably well.
Going by value alone misses some things, though. A small value sitting at exactly the wrong bottleneck can bring everything downstream crashing when it's cut. So sometimes the actual effect of removing a given connection on the final answer gets measured directly, and choices get made from there. How much can safely be cut also differs layer by layer — earlier layers generally have little room to spare, while later ones tend to have more.
Cut a little, then retrain
Cut a large chunk all at once and the answer noticeably gets worse. So instead, cut a little, then retrain briefly on what's left so it can fill in the gap. Repeat this two-step cycle several times over.
Even cutting the same total amount, this approach loses far less performance. The connections that remain gradually pick up a share of what the removed ones used to do. Cut everything up front with no retraining at all, on the other hand, and the answer tends to collapse sharply once you cross a certain line.
Scattered pruning versus wholesale pruning
Cut connections one at a time and what's left ends up scattered all across the table. The table's shape stays the same even with plenty of cells gone empty, so an ordinary processor still scans the whole table regardless. If the emptied cells get tidied up and stored separately, storage shrinks, but speed often doesn't pick up nearly as much as hoped.
That's why wholesale removal — cutting entire rows or blocks at once — often gets used alongside it. Removing one whole unused neuron, or one whole filter in an image model that barely ever fires, are examples. The table genuinely shrinks this way, so the computation genuinely shrinks too. In exchange, performance wobbles more per cut, so this kind of cutting has to be done more cautiously.
How it differs from quantization
Both make a model lighter, but they touch different things. Quantization doesn't throw away a single number — it only coarsens the scale those numbers are written on. Pruning leaves the scale untouched and instead reduces how many numbers there are.
In menu-board terms: quantization is leaving every item on the board and just rounding all the prices to the nearest dollar; pruning is striking the never-ordered items off the board entirely. Used together they make the model lighter still, so they're often combined — cut what's unused first, then coarsen the scale of what's left. But when the two errors stack up, the result can wobble more, so the order and the amount change the outcome.
More precisely
Pruning removes low-contribution weights or neurons from a neural network to make it sparse. It's mostly done after training finishes, but it can also be done a little at a time during training. It splits into unstructured pruning, which removes individual weights, and structured pruning, which removes entire channels or layers — the latter is more likely to translate into a real speedup. There's also a widely discussed idea that a well-performing smaller skeleton already sits hidden inside a large model from the start.
The analogy breaks down in a place too. An item struck from the menu board is genuinely gone — customers can't order it. A connection cut in the scattered way, though, is still sitting there as a zero inside the table, so squeezing storage down for real needs a separate cleanup step. And a food stall judges popularity by order count, while a model's connections get judged by how much they shake the result, not how often they fire — some connections that almost never activate turn out to be decisive on the rare occasion they do. Whichever way you cut, run the evaluation again afterward to check what changed.
Try it yourself
- Netron (model structure viewer) ailearn.space Open a model file and you can see the size of each layer's weight table. Knowing which layers hold most of the numbers gives you a feel for the big layers pruning would target
- TensorFlow Playground ailearn.space Train it, then remove neurons and inputs one at a time. Some barely change the result; others break it instantly
Common misconceptions
It's easy to think pruning just shrinks a model carelessly, but actually it's a careful process — picking the lowest-contributing connections first and cutting in a way that loses as little performance as possible.
It's easy to think removing connections speeds things up immediately, but actually cutting them scattered leaves the table's shape unchanged, so on an ordinary processor speed barely improves.
It's easy to think pruning and quantization are the same thing, but actually one reduces the count of numbers and the other coarsens the scale they're written on — two different moves entirely.
One-line summary
Pruning picks out connections that barely contribute to the answer, cuts them away, and retrains what's left to make a model lighter.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-17