Key points
- Vanishing gradient is what happens when the signal telling a network what to fix shrinks more and more on its way back down through the layers, until it barely reaches the layers near the input.
- The signal gets passed along by multiplication at every layer it crosses. Multiply a number smaller than one together with itself enough times and it collapses toward zero fast.
- With the early layers stuck nearly frozen, stacking a network deeper can even make results worse, not better.
- Swapping the activation function, keeping value scales even, and shortcuts that skip past layers have solved a large share of this.
- Go the other direction and signals that only ever grow at each layer cause the opposite problem: values exploding instead.
The analogy
Picture a line of classmates playing telephone. The person at the back whispers "start again from row three" into the ear of the person in front. There's one rule: pass along only half the loudness of what you heard.
Five people in and the message is already a mumble; by the tenth, all that reaches the next ear is a breath of sound. The person at the very front never hears what needs fixing and just stays standing exactly the way they started. Everyone toward the back keeps getting refined while the front stays frozen — that's a vanishing gradient.
The longer the line, the worse this gets. Add more people and the front just gets quieter still.
In detail
The signal flows backward, from output to input
When a model produces an answer, computation flows from input toward output. The signal saying "here's what needs fixing" flows the opposite way, from output back toward input. How wrong the answer was is only knowable once you reach the very end.
So the layers closest to the output receive this signal first, and most clearly. The layers closest to the input only receive it after it's crossed every layer in between. It's the same order as a whisper that starts at the back of the line and has to travel all the way to the front.
It gets multiplied at every layer it crosses
Crossing one layer, the signal's size gets adjusted based on that layer's own values and the character of its activation function. The catch is that this adjustment is multiplication, not addition. If every layer cuts the signal in half, by the tenth layer it's already down to under a thousandth of where it started.
The S-shaped activation function used heavily in the past was especially bad about this — its output flattens out the moment the input gets even moderately large or small. In that flat stretch, what gets passed along is tiny, so the signal shrinks noticeably after stacking just a handful of layers. Go the other way and a layer-by-layer multiplier bigger than one lets the signal balloon out of control instead, sending values spiking.
What happens when the early layers stall out
The early layers are where the most basic features of text or images get picked up. If those layers can't learn, everything stacked above them has to keep working with raw, unrefined material. Stack more layers to try to boost performance, and when the vanishing is severe you can end up with results worse than a shallower model. Separately, even with the signal protected, a deep network can still learn less than hoped, and shortcut connections help with that problem too.
The symptoms tend to look like this: training runs on for a long time but the error stops dropping at some point, and the numbers in the early layers barely move from wherever they started out random. In models that process long sequences in order, this can also show up as information from early in the sequence never making it through to later positions.
Ways to protect the signal
The biggest turning point was the shortcut that skips across layers — instead of the signal shrinking as it crosses one layer at a time, a separate path was carved out that lets it jump several layers at once. It's the equivalent of giving the person at the back of the telephone line a direct line straight to the person at the front.
Using an activation function that doesn't flatten out helped enormously too. A function that passes a value through at full size wherever the value is positive keeps the multiplier from shrinking at all. On top of that came techniques for keeping value scales even layer to layer, and care in not starting the numbers off too large or too small. In models that read in order, gate structures that decide what to remember and what to let go of were added to help the signal survive longer.
More precisely
Backpropagation is a chain computation that multiplies each layer's local derivative together in sequence. If the value multiplied in at each layer averages below one, it shrinks exponentially with the number of layers; if it averages above one, it grows exponentially for the same reason. The derivative of the sigmoid function tops out at exactly 0.25 even at its steepest, so as long as the weights don't exceed 1 in size, the signal shrinks by at least a factor of four each time it passes through one.
The analogy breaks down in one place. With a whispered message, the trouble is that the content drifts into something completely different, while the heart of a vanishing gradient is that the size shrinks too far. It's more accurate to say the signal hasn't vanished entirely — it's just become too small to move the needle during a training step.
Not every path shrinks equally, either. Some routes survive while others die off, so a single layer can end up with a mix of parts that are still learning and parts that have stalled.
Try it yourself
Common misconceptions
It's easy to think stacking layers deeper always makes a model smarter, but actually if the signal never reaches the early layers, results can come out worse than a shallower model.
It's easy to think this is an old problem that's fully solved now, but actually it's been eased by a growing toolkit of fixes, not eliminated, and it still matters for very deep networks or very long inputs.
It's easy to think raising the learning rate solves it, but actually that lets already-large signals grow even larger, which can make values spike and training more unstable.
One-line summary
Vanishing gradient is when the signal telling a network what to fix gets multiplied down at every layer until it barely reaches the early layers, and learning there slows to a crawl.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-17