Key points
- An optimizer takes the gradient and decides how to actually move the values. The downhill direction to start from comes from the gradient and the base step size from the learning rate; the optimizer reworks both into the actual step.
- The first trick is momentum. Keep heading the same way and the speed from past steps carries forward into a bigger stride.
- The second trick is giving each direction its own stride. A value that has been receiving large gradients gets its stride shrunk, and one receiving small gradients gets its stride grown, which evens out the step size across values.
- Together these cut down the wasted back-and-forth in a narrow, steep-walled valley, so the same amount of time gets you much further down.
- Switching optimizers doesn't let you skip picking a learning rate. The baseline step size is still a person's call.
The analogy
In a relay race, if the next runner stands still at the exchange zone waiting for the baton, the team's time takes a real hit — starting from a dead stop and building speed back up costs seconds. So instead, that runner starts moving before the baton even arrives, matching the incoming runner's speed and carrying it forward.
The footwork changes with the track, too. On a curve, the body gets pushed outward, so the strides shorten and tighten up; on the straightaway, the strides stretch out long. Same track, different footwork depending on which stretch you're on.
That's exactly what an optimizer does: it carries forward the speed from the last step, and it varies the stride depending on the stretch. The big picture of which way is downhill comes from the gradient, and the optimizer handles the style of running that path. That style is also why the direction of the actual step shifts a little.
In detail
Direction alone is wasteful
The gradient tells you which way is downhill from here. But moving strictly along that direction, with nothing else, wastes effort — especially in a valley that's steep on one wall and gentle on the other.
In a spot like that, you bounce hard off the steep wall and back again, while barely creeping forward along the gentle direction that actually leads down the valley. You're supposed to be following the valley floor, but instead you zigzag off the walls. Lots of steps, not much actual descent.
An optimizer doesn't use the raw gradient as-is. It remembers past steps and blends them with the current gradient to work out how far to actually move.
Momentum smooths out the wobble
The first trick is carrying forward a fraction of the last step's movement. Keep heading the same way and that carried-over share stacks up, so the stride keeps growing; a direction that flips back and forth every time cancels itself out and fades.
Applied to the valley, the side-to-side bouncing off the walls dies down, and only the forward motion down the valley survives, picking up speed. Momentum can even carry the model past a shallow dip it would otherwise settle into, cutting the risk of getting stuck somewhere mediocre.
The fraction carried forward is usually set to keep most of the previous share. Set it too high and the model keeps drifting in the old direction long after it should have turned; set it too low and it's barely different from having no momentum at all.
Different strides for different directions
The second trick is tracking a separate stride for every value. Keep a record of how large the gradients arriving at each value have been, then shrink the stride for values that received large ones and grow it for values that received small ones.
This helps most with a feature that rarely shows up. A value tied to something that appears only once in a while gets few chances to update, so growing its stride makes those rare chances count for more. Adam, a widely used optimizer, combines both tricks — momentum and per-direction strides — together.
The baseline stride is still a person's call
An optimizer adjusting the stride on its own doesn't make the learning rate unnecessary. What it adjusts is a multiplier; the baseline that multiplier gets applied to is still something a person sets. Set that baseline too high, and even good technique won't stop the bouncing.
That's why practice usually starts from a widely used optimizer as the default and tunes the learning rate first. Getting the learning rate right often shakes up the outcome more than swapping optimizers does.
More precisely
An optimizer is a rule that takes the gradient plus a record of past updates and computes this step's update amount. The simplest form is plain stochastic gradient descent (SGD), which just multiplies the gradient by the learning rate. From there came a version that adds a momentum term carrying forward past movement, and one that divides the stride using the accumulated size of past gradients per value. The Adam family, which combines momentum with per-direction strides, is the most widely used today.
The comparison breaks down in places too. A relay track's curves and straightaways are visible, but training has no way to know in advance which direction is steep. It can only guess from the record of how large the gradients have been so far. And a runner changes one stride depending on the stretch, while an optimizer uses a different stride for each value even at the same instant — like countless legs each setting their own stride. And an optimizer that uses momentum or per-direction strides keeps a separate record like this for every single value; with Adam, that takes two extra sets of memory the size of the model's values.
Try it yourself
Common misconceptions
It's easy to think switching to a better optimizer means the learning rate doesn't matter anymore, but actually the baseline step size is still a person's call, and that value often swings the outcome a lot.
It's easy to think that if swapping optimizers lowers the training error, it must do better on new data too, but actually it can't change the landscape itself — it only changes which path gets taken down the same landscape and where it ends up stopping. So a lower training error can still come with a worse score on new data.
It's easy to think the newest optimizer is always best, but actually an older, simpler method often fits the data and model better, depending on the situation.
One-line summary
An optimizer takes the downhill direction, carries forward its past speed, and gives each direction its own stride — it's the technique behind how the values actually move.
Spotted an error or have a better analogy? Suggest an edit · Last updated 2026-10-17