Key points
- Convergence is the state where the values barely change and settle in place, no matter how much longer training runs.
- The improvement is big at first and shrinks over time, which is why the chart drops steeply, then flattens out.
- Calling it "done" is often settled by comparing how much things changed over the last several rounds against a set threshold.
- If the values never settle and keep bouncing, or even keep growing, that's non-convergence, and too large a step is the usual cause.
- Convergence just means training has stopped. It doesn't mean the result landed somewhere good.
The analogy
Fill a big barrel with muddy water and let it sit, and the heavier grit sinks to the bottom first. Right after the water goes in, the turbidity gauge needle drops fast. After about an hour the drop noticeably slows, and by half a day, the reading taken that morning and the reading taken that evening are nearly identical. Wait another full day and the needle stays right where it is — because everything that was going to settle has already settled.
At that point, the person watching decides waiting any longer is pointless and pours off the clear water on top. Wherever the needle stopped is the best this barrel can do. Training reaching convergence is the same scene. The numbers stop changing no matter how much longer it runs, so it's time to stop.
In detail
Big changes first, small ones later
Training repeats the same cycle: measure how wrong things are, then move the values by that much. Early on, the values sit somewhere off, so the error is large and there's a long way to travel. That's why loss drops sharply in the first several rounds.
Once the values settle somewhere reasonable, things change. The remaining error is small, so the signal telling the values to move gets small too. The step size effectively shrinks on its own, which is why later rounds bring thinner and thinner gains — same as how the heavy grit settles first, and the fine particles left behind barely sink at all.
How do you know it's done
The call to stop doesn't have to rest on eyeballing a chart alone; a threshold is often set in advance. If the loss over the last several rounds has dropped by less than that threshold, it counts as converged. Checking several rounds in a row instead of judging from just one guards against a single round happening to go quiet by chance. A loss that looks flat can also start falling again much later, or drop another notch the moment the step size is reduced, so the required number of rounds is sometimes set generously.
In practice, it's not just training loss that gets watched — the score on data the model has never seen gets tracked alongside it. If training loss keeps inching down while the score on new data stops improving, whatever's left of that drop is mostly memorization. So training is sometimes stopped at the point where the score on new data stops improving, which means stopping before the training loss has fully settled.
When it never settles, just keeps shaking
Sometimes the values simply refuse to settle. Loss drops, then spikes back up, over and over — or it just keeps growing until the whole calculation breaks. It's the same as stirring the barrel hard instead of leaving it still: even the grit that had already sunk gets stirred back up.
The most common cause is too large a step. It's heading toward the target, but overshoots it every time and lands on the other side instead. The fix is shrinking the step, or shrinking it progressively as training goes on. Scaling values to a consistent range beforehand, and clipping any signal that's grown too large, are both common fixes too.
Settled isn't the same as good
Convergence only tells you "it's not changing anymore." It doesn't say whether that spot is any good. The water in the barrel going no clearer doesn't mean it is clean enough to drink. If the leftover turbidity is above the standard, you need to add a treatment chemical or filter it once more.
Training is the same. If the score is still low after convergence, running it longer isn't the answer — something else needs to change. More data, a bigger model, different starting values, then run it again. Stopping at a spot that's lower than its surroundings but not the lowest has its own name — a local minimum — though in networks with a huge number of parameters, getting trapped in a bad local minimum turns out to be rarer than you'd expect; a low score more often comes from too little data or too little model capacity. So when deciding training is "done," check both whether the values have stopped and whether that score is good enough.
More precisely
Convergence is the state where, as iterations continue, the change in loss or in the parameters drops below a set threshold and effectively settles at one point. With mini-batch training, each batch draws on different data, which mixes noise into the signal, so the values never fully stop — they keep wobbling faintly within a narrow range. That's why the actual call isn't "it became exactly the same," but "the amount of change dropped below the threshold."
The comparison breaks down in places too. Run the same water through a barrel again and again, and it ends up at roughly the same turbidity every time — but neural network training tends to land somewhere slightly different each run, depending on the starting values and the order the data was shown in. That's why two runs with identical settings still finish with slightly different final scores. And settling happens on its own given enough time, while training with too large a step size never settles no matter how long you wait.
Try it yourself
Common misconceptions
It's easy to think convergence means the correct answer has been reached, but actually it just means things stopped changing — the score at that spot can still be low.
It's easy to think running longer means converging more, but actually once things go quiet, running longer just burns time and money, and the score on new data can even get worse.
It's easy to think loss has to reach zero to count as converged, but actually convergence only means the values stop changing; zero isn't the benchmark. Noise in the data or limits of the model often leave loss settling above zero, while a large model can memorize its training data and push loss close to zero.
One-line summary
Convergence is the state where values stop changing no matter how much longer training runs — it tells you when to stop, but not whether that spot is the best one.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-17