Key points
- Layer normalization is a device that recalibrates the scale of the numbers every time a layer is passed through.
- What it calibrates against is the bundle of numbers held at a single position — it never looks at a neighboring example or a different sentence.
- After recalibrating, two dials get attached — one to stretch the range back out, one to shift it. How far to stretch is something the model decides through training.
- Without this device, as layers get deeper the numbers tend to blow up or shrink toward nothing, and training becomes unstable.
- Today's models mostly place it before entering a layer, rather than after.
The analogy
Picture one student's row on a practice-exam score report. Even for the same student, on an easy-exam day the whole row runs high and on a hard day it runs low, so raw scores alone make it hard to see right away which subject stands out and which one lags. So imagine rewriting that row using only that student's own scores: take their average as the reference point and even out the spread to a consistent width.
The same thing happens inside a neural network. Every time a signal passes through a layer, the numbers grow or shrink unpredictably. Some spots spike up into the hundreds; others cluster near zero. Hand this straight to the next layer, and one spiking spot can swamp the entire judgment.
Layer normalization is like reissuing the score report every time a layer is passed through. Just like one student's row, it uses only the numbers at that position: it shifts their average to the center, evens out the spread to a consistent width, and hands the result to the next layer.
In detail
When the scale drifts, training gets shaky
The deeper a network gets, the bigger this problem becomes. If one layer has a habit of inflating numbers to one-and-a-half times their size, that habit compounds over twenty layers into something thousands of times the original. Go the other way, and shrinking a little at every layer leaves almost nothing left by the last layer.
Training works by sending error backward from the last layer toward the first, and the value traveling backward goes through the exact same multiplication. So the earliest layers either receive no signal at all, or a signal so large they can't handle it. If the earliest layers can't learn, the whole point of stacking deep disappears.
Layer normalization evens out the scale of the numbers each layer receives, every single time. That way the calculation inside a layer always starts from numbers in a similar range, and combined with skip connections, a network can stack dozens of layers deep and still train stably.
The reference point is just one position
The "one position" here means the bundle of numbers a single piece in a sentence is carrying. A language model carries thousands of numbers for every single piece. Layer normalization computes its average and spread using only those thousands of numbers, nothing outside them.
Not looking sideways matters. However many sentences get processed together, however long or short a sentence is, each position calibrates using only its own numbers. That's why it behaves exactly the same whether you feed in a single sentence to get an answer or a whole batch during training.
This trait is especially useful for handling text of wildly varying lengths. Approaches that average across multiple examples get thrown off when short and long pieces of text get mixed together, since the baseline keeps shifting.
Two dials to undo it with
Forcing the scale back into a fixed range can erase differences a model worked hard to build. So two dials get attached right after normalizing. One stretches or shrinks the range back out; the other shifts the whole thing up or down.
These dials aren't set by a person — they get set through training. If keeping a large difference in a particular slot of the bundle turns out to help, the model turns that slot's dial up on its own. Every piece shares the same dials. Normalization isn't throwing all the information away — it's leveling the starting line and leaving how far to stretch entirely up to the model. What is lost is how large the whole bundle originally was.
Before the layer, or after?
Early designs attached normalization right after a layer finished its computation. This placement tended to be unstable early in training, so a slow warm-up with a tiny learning rate before ramping up was practically mandatory, and it got harder to manage the deeper the layers went.
The placement widely used now is the opposite. It normalizes first, before entering a layer, and the layer's result gets added onto the untouched value that traveled through a skip connection. This keeps a clean channel running from start to finish, letting even very deep models train reliably.
Whether training succeeds or fails can hinge on where this exact same device gets placed. That where matters more than the calibration itself is one of the lessons learned from working with deep models.
More precisely
Layer normalization computes the average and standard deviation within one example's bundle of feature values, subtracts the average, divides by the standard deviation, then rescales and shifts the result back out using a separately trained scale and shift for each slot. Batch normalization, which groups values across multiple examples at matching positions instead, runs in exactly the opposite direction. Some recent models skip the subtract-the-average step entirely and use a lighter variant that only rescales, since it barely hurts performance while cutting computation.
The analogy breaks down in a couple of places. A real score report is converted by comparing against other test takers, but layer normalization never looks at other examples and only calibrates the numbers within one position. Converting a score report happens once, for a human reader's benefit, but layer normalization happens over and over — once per layer, for every single piece of an answer. A score report also tells you a rank within a subject, but normalization never ranks anything — the subtract-the-average and rescale step leaves the order and spacing ratios of the numbers alone and only changes the scale. The dials after that are adjusted separately for each slot. And though the name contains "normalization," its purpose is nothing like the regularization used to fight overfitting. This one is a computational stability device, meant purely to keep training from falling apart.
Try it yourself
Common misconceptions
It's easy to think this is a one-time preprocessing step done before data goes in, but actually it happens fresh every single time a layer is passed through, inside the model itself.
It's easy to worry that calibrating the scale destroys all the information, but actually, in the calibrating step, the order and relative spacing of the numbers in the bundle stay intact, and the trained dials can reset a useful spread and position. What does get erased is how large the whole bundle originally was.
It's easy to think this is a device for fighting overfitting, but actually it's a stability device that keeps numbers from blowing up or vanishing in deep layers — a completely different job.
One-line summary
Layer normalization recalibrates the scale of the numbers going into or coming out of each layer every time, letting a deeply stacked network train without falling apart.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-17