Residual Connection

A layer only has to learn the difference

An ordinary layer takes what it receives and turns the whole thing into a new value. Attach a shortcut to that layer and its job changes. What the layer produces is no longer a finished result — it's the difference between the original value and the target. That's where the name residual comes from: what's left over.

Only having to produce a difference makes the layer's job much easier. If the current value is already good enough, the layer can output something close to zero and add almost nothing. That matters more than it sounds like it should. Without a shortcut, even doing nothing is something a layer has to learn to imitate precisely by tuning its own knobs just right.

It also becomes possible to stack a hundred layers and let only thirty of them actually do meaningful work. The rest simply pass along what they received.

Adding requires matching sizes

For the value that arrives by the shortcut and the value the layer produces to be combined, the two have to be the same shape. Bundles of numbers with different lengths can't be added together. That's why the stretch of network wrapped by a shortcut is usually designed so the size coming in matches the size going out.

Where the size does need to change, a very light transformation gets slipped into the shortcut path itself to match it up. In image models, the spot where the picture shrinks and the number of channels grows is one such place. Even so, the rule is to keep that transformation as minimal as possible — a shortcut that gets complicated stops being a shortcut.

Without a shortcut, deeper meant worse

It seems natural to assume that stacking layers deeper would boost capacity and therefore performance. In practice the opposite happened. Push the layer count from twenty to fifty and results got worse, and not just on new data — even on data the model had already trained on.

This wasn't the same as memorizing too hard. A deep model should have been able to match a shallower model's performance at minimum just by copying its answers, but it couldn't even manage that. The problem wasn't a lack of capacity — training itself simply wasn't working. Once shortcuts arrived, that wall came down, and models with well over a hundred layers began performing better in practice.

The learning signal rides the shortcut back down too

Learning works by measuring how far off the top-level answer was and sending that signal back down toward the earlier layers. Each time this signal crosses a layer, it gets caught up in that layer's own computation and shrinks or grows a little. Across dozens of layers, by the time it reaches the earliest ones, the signal has nearly vanished.

With a shortcut in place, the returning signal rides that same path down. Since there's a route that bypasses the layer computation entirely, the signal arrives at the early layers close to its original strength. That's the reason the earliest layers of a deep model can finally learn properly.

Built into nearly everything today

Shortcuts first succeeded in a big way inside image recognition models, but by now they're a default part of almost every architecture. Open up a single block inside a language model and you'll find two shortcuts: one right after the attention step, and another right after the layer that follows it.

So instead of getting completely overwritten as it passes through the layers, the value keeps one thick, unbroken stem running straight from bottom to top. Each layer just adds its own small contribution to that stem. Layer normalization tends to show up as a constant companion for exactly this reason — if every layer only ever adds, the values would otherwise keep growing larger and larger.