What is Vanishing Gradient? — one analogy

1The analogy

Picture a line of classmates playing telephone. The person at the back whispers "start again from row three" into the ear of the person in front. There's one rule: pass along only half the loudness of what you heard.

Five people in and the message is already a mumble; by the tenth, all that reaches the next ear is a breath of sound. The person at the very front never hears what needs fixing and just stays standing exactly the way they started. Everyone toward the back keeps getting refined while the front stays frozen — that's a vanishing gradient.

The longer the line, the worse this gets. Add more people and the front just gets quieter still. Nobody at the back did anything wrong — the rule itself, applied over and over, is what erases the message before it can travel the full distance.

2In detail

The signal flows backward, from output to input

When a model produces an answer, computation flows from input toward output. The signal saying "here's what needs fixing" flows the opposite way, from output back toward input. How wrong the answer was is only knowable once you reach the very end.

So the layers closest to the output receive this signal first, and most clearly. The layers closest to the input only receive it after it's crossed every layer in between. It's the same order as a whisper that starts at the back of the line and has to travel all the way to the front.

It gets multiplied at every layer it crosses

Crossing one layer, the signal's size gets adjusted based on that layer's own values and the character of its activation function. The catch is that this adjustment is multiplication, not addition. If every layer cuts the signal in half, by the tenth layer it's already down to under a thousandth of where it started.

The S-shaped activation function used heavily in the past was especially bad about this — its output flattens out the moment the input gets even moderately large or small. In that flat stretch, what gets passed along is tiny, so the signal shrinks noticeably after stacking just a handful of layers. Go the other way and a layer-by-layer multiplier bigger than one lets the signal balloon out of control instead, sending values spiking.

What happens when the early layers stall out

The early layers are where the most basic features of text or images get picked up. If those layers can't learn, everything stacked above them has to keep working with raw, unrefined material. Stack more layers to try to boost performance, and you can end up with results worse than a shallower model — this is exactly where that happens.

The symptoms tend to look like this: training runs on for a long time but the error stops dropping at some point, and the numbers in the early layers barely move from wherever they started out random. In models that process long sequences in order, this can also show up as information from early in the sequence never making it through to later positions.

Ways to protect the signal

The biggest turning point was the shortcut that skips across layers — instead of the signal shrinking as it crosses one layer at a time, a separate path was carved out that lets it jump several layers at once. It's the equivalent of giving the person at the back of the telephone line a direct line straight to the person at the front.

Using an activation function that doesn't flatten out helped enormously too. A function that passes a value through at full size wherever the value is positive keeps the multiplier from shrinking at all. On top of that came techniques for keeping value scales even layer to layer, and care in not starting the numbers off too large or too small. In models that read in order, gate structures that decide what to remember and what to let go of were added to help the signal survive longer.

3More precisely

Backpropagation is a chain computation that multiplies each layer's local derivative together in sequence. If the value multiplied in at each layer averages below one, it shrinks exponentially with the number of layers; if it averages above one, it grows exponentially for the same reason. The derivative of the S-shaped activation function tops out around 0.25 even at its steepest, so stacking that function alone shrinks the signal by at least a factor of four at every layer.

The analogy breaks down in one place. A whispered message not only gets quieter but also drifts into something completely different in content, while a vanishing gradient mostly keeps its direction intact and only shrinks in size. It's more accurate to say the signal hasn't vanished entirely — it's just become too small to move the needle during a training step.

Not every path shrinks equally, either. Some routes survive while others die off, so a single layer can end up with a mix of parts that are still learning and parts that have stalled.

The opposite failure, exploding values, comes from the exact same chain of multiplication running the other direction, which is why the two are usually discussed as a matched pair rather than separate problems.

4Try it yourself

5Common misconceptions

It's easy to think stacking layers deeper always makes a model smarter, but actually if the signal never reaches the early layers, results can come out worse than a shallower model.
It's easy to think this is an old problem that's fully solved now, but actually it's been eased by a growing toolkit of fixes, not eliminated, and it still matters for very deep networks or very long inputs.
It's easy to think raising the learning rate solves it, but actually that lets already-large signals grow even larger, which can make values spike and training more unstable.

7One-line summary

In shortVanishing gradient is when the signal telling a network what to fix keeps shrinking by half at every layer until it never reaches the early layers, and learning there simply stalls.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02