Backpropagation

Forward once, backward once

One step of training is a round trip. Data goes in first and passes through the layers in order to produce an answer — that's pushing the car into the wash. The answer gets compared to the correct one and the error gets measured.

Then comes the backward leg. The error is handed to the last layer, which works out how much its own values contributed to it and passes the remaining share to the layer before it. That handoff continues all the way to the first layer. Once both legs finish once, every value in the model carries a note: "move this way, by this much."

The in-between results produced on the forward pass have to be kept, not thrown away — the backward pass needs them again. That's the main reason training eats far more memory than just running a finished model.

Why start from the back

You could imagine checking from the front instead: tweak one value slightly and measure how much the result changes. But every time one value changes, the whole car wash has to run again. With hundreds of millions of values, that's hundreds of millions of runs.

Working backward eliminates that waste. Once you know the later stage's share, the earlier stage's share falls out in a handful of multiplications. One sweep from back to front produces every value's share at the same time. That's the only reason training time stays manageable no matter how many layers get stacked.

Blame first, then move

Backpropagation's job ends at splitting the blame. Actually moving the values based on that blame belongs to gradient descent. How far to move comes from the learning rate; the style of moving comes from the optimizer. The roles are split up this cleanly.

The blame gets recalculated fresh every time a batch of data is checked. The same value might get told to go up this time and down the next, depending on what data it saw. That's why training doesn't react hard to any single share — it only moves a little at a time.

The blame fades the further back it travels

Working backward multiplies a value at every layer it passes through. If that multiplier is smaller than one, the share shrinks with every layer, and by the time it reaches the first layer it's nearly zero. That's how the vanishing gradient happens — the front layers stop learning and stall out.

If the multiplier is large instead, the share balloons the further back it goes, and values blow up. That's why shortcuts that skip past layers, and devices that rescale values at every layer, are so widely used.

Deep models went unlearnable for years because of exactly this problem. The backward procedure itself had been known for a long time — it just took devices that could keep the blame alive all the way to the front before layers could be stacked freely.