Dropout

A model leaning on just a few units

A model holds a huge number of units passing values back and forth. Over a long training run, a handful of them can pair up into one particular combination that ends up carrying almost the whole answer, while the rest settle for backing it up at best.

A model like this scores well on the data it trained on. But when new data doesn't fit that combination, there's nothing else to fall back on, and the mistakes are large. A structure leaning that hard on one side falls apart the moment the data shifts even slightly.

Dropout breaks up this leaning during training itself. Since a different set of units drops out every step, no particular combination ever gets the chance to set in. Training drifts toward a state where every unit builds a signal that's useful on its own.

A new draw every step

Which units rest isn't decided in advance. Every time a batch of data is processed, the units are redrawn on the spot. A unit picked to rest holds its value at zero for that step, and it's skipped when credit gets divided up on the way back too.

That means every step trains a slightly different model. With thousands of units, the number of possible combinations is effectively endless. The finished model ends up close to a blend of countless slightly different models trained this way — a trick that imitates, inside a single model, the old rule of thumb that averaging several models' answers makes for a steadier one.

How much rest is enough

The rate is set by a person. Resting close to half the units is a common setting, and layers with fewer units get a much lower rate. Too high, and the units left standing can't carry the load, so training struggles. Too low, and it fails to break up the leaning, so setting it barely does anything.

Layers can carry different rates too — heavier where units are packed densely, lighter near where values first come in. On training runs with a huge amount of data, it's sometimes skipped entirely, since the same data is never repeated enough to get memorized.

It's worth knowing that dropout slows training down a little too. Learning with part of the model missing every step means more turns of the wheel to reach the same score. Looking only at scores on the training data, the version without dropout always looks better — but measure it on data set aside instead, and the order flips.

Everything switched on when it's used

Once training ends and the model actually goes to work, nothing rests anymore. Getting a different answer to the same question every time would be no good. But simply switching everything back on causes a problem of its own: during training, values always flowed with part of the model missing, so now that everything is on, the values come out that much larger.

That's why a matching adjustment rides along with it. A common approach scales up the surviving units' values ahead of time, during training itself. Do that, and no correction is needed at use time — the size of the values already lines up with what training expected.