Convergence
When training stops improving and settles in place
- Convergence is the state where the values barely change and settle in place, no matter how much longer training runs.
- The improvement is big at first and shrinks over time, which is why the chart drops steeply, then flattens out.
- Calling it "done" isn't a matter of eyeballing the chart — it's decided by comparing how much things changed over the last several rounds against a set threshold.
- If the values never settle and keep bouncing, or even keep growing, that's non-convergence, and too large a step is the usual cause.
- Convergence just means training has stopped. It doesn't mean the result landed somewhere good.
Contents
1The analogy
Fill a settling tank at a water treatment plant with muddy water and let it sit, and the heavier grit sinks to the bottom first. Right after the water goes in, the turbidity gauge needle drops fast. After about an hour the drop noticeably slows, and by half a day, the reading taken that morning and the reading taken that evening are nearly identical. Wait another full day and the needle stays right where it is — because everything that was going to settle has already settled.
At that point, the operator decides waiting any longer is pointless and sends the water on to the next stage. Wherever the needle stopped is the best this tank can do. Training reaching convergence is the same scene. The numbers stop changing no matter how much longer it runs, so it's time to stop.
2In detail
Big changes first, small ones later
Training repeats the same cycle: measure how wrong things are, then move the values by that much. Early on, the values sit somewhere off, so the error is large and there's a long way to travel. That's why loss drops sharply in the first several rounds.
Once the values settle somewhere reasonable, things change. The remaining error is small, so the signal telling the values to move gets small too. The step size effectively shrinks on its own, which is why later rounds bring thinner and thinner gains — same as how the heavy grit settles first, and the fine particles left behind barely sink at all.
How do you know it's done
The call to stop isn't made by eyeballing a chart. A threshold gets set in advance. If the loss over the last several rounds has dropped by less than that threshold, it counts as converged. Checking several rounds in a row instead of judging from just one guards against a single round happening to go quiet by chance.
In practice, it's not just training loss that gets watched — the score on data the model has never seen gets tracked alongside it. If training loss keeps inching down while the score on new data stops improving, whatever's left of that drop is mostly memorization. So the stopping point gets picked where both lines go quiet together.
When it never settles, just keeps shaking
Sometimes the values simply refuse to settle. Loss drops, then spikes back up, over and over — or it just keeps growing until the whole calculation breaks. It's the same as stirring the settling tank hard instead of leaving it still: even the grit that had already sunk gets stirred back up.
The most common cause is too large a step. It's heading toward the target, but overshoots it every time and lands on the other side instead. The fix is shrinking the step, or shrinking it progressively as training goes on. Scaling values to a consistent range beforehand, and clipping any signal that's grown too large, are both common fixes too.
Settled isn't the same as good
Convergence only tells you "it's not changing anymore." It doesn't say whether that spot is any good. The settling tank going no clearer doesn't mean the water is clean enough to drink. If the leftover turbidity is above the standard, the chemicals or the process need to change.
Training is the same. If the score is still low after convergence, running it longer isn't the answer — something else needs to change. More data, a bigger model, different starting values, then run it again. Getting stuck in a bad spot has its own name — a local minimum — and any call that training is "done" always checks both of these things together.
3More precisely
Convergence is the state where, as iterations continue, the change in loss or in the parameters drops below a set threshold and effectively settles at one point. With mini-batch training, each batch draws on different data, which mixes noise into the signal, so the values never fully stop — they keep wobbling faintly within a narrow range. That's why the actual call isn't "it became exactly the same," but "the amount of change dropped below the threshold."
The comparison breaks down in places too. Run the same water through a settling tank again and again, and it ends up at roughly the same turbidity every time — but training lands somewhere different each run, depending on the starting values and the order the data was shown in. That's why two runs with identical settings still finish with slightly different final scores. And settling happens on its own given enough time, while training with too large a step size never settles no matter how long you wait. A settling tank also has just one thing to watch — turbidity — while training tracks several signals side by side, and calling something converged usually means all of them have gone quiet together, not just one.
4Try it yourself
5Common misconceptions
It's easy to think convergence means the correct answer has been reached, but actually it just means things stopped changing — the score at that spot can still be low.
It's easy to think running longer means converging more, but actually once things go quiet, running longer just burns time and money, and the score on new data can even get worse.
It's easy to think loss has to reach zero to count as converged, but actually noise mixed into the data means loss normally settles above zero — that's expected.
7One-line summary
In shortConvergence is the state where values stop changing no matter how much longer training runs — it tells you when to stop, but not whether that spot is the best one.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02