What is Optimizer? — one analogy

1The analogy

In a relay race, if the next runner stands still at the exchange zone waiting for the baton, the team's time takes a real hit — starting from a dead stop and building speed back up costs seconds. So instead, that runner starts moving before the baton even arrives, matching the incoming runner's speed and carrying it forward.

The footwork changes with the track, too. On a curve, the body gets pushed outward, so the strides shorten and tighten up; on the straightaway, the strides stretch out long. Same track, different footwork depending on which stretch you're on.

That's exactly what an optimizer does: it carries forward the speed from the last step, and it varies the stride depending on the stretch. Which way to run is already decided — the optimizer only handles the style of running that path.

2In detail

Direction alone is wasteful

The gradient tells you which way is downhill from here. But moving strictly along that direction, with nothing else, wastes effort — especially in a valley that's steep on one wall and gentle on the other.

In a spot like that, you bounce hard off the steep wall and back again, while barely creeping forward along the gentle direction that actually leads down the valley. You're supposed to be following the valley floor, but instead you zigzag off the walls. Lots of steps, not much actual descent.

An optimizer doesn't use the raw gradient as-is. It remembers past steps and blends them with the current gradient to work out how far to actually move.

Momentum smooths out the wobble

The first trick is carrying forward a fraction of the last step's movement. Keep heading the same way and that carried-over share stacks up, so the stride keeps growing; a direction that flips back and forth every time cancels itself out and fades.

Applied to the valley, the side-to-side bouncing off the walls dies down, and only the forward motion down the valley survives, picking up speed. Momentum can even carry the model past a shallow dip it would otherwise settle into, cutting the risk of getting stuck somewhere mediocre.

The fraction carried forward is usually set to keep most of the previous share. Set it too high and the model keeps drifting in the old direction long after it should have turned; set it too low and it's barely different from having no momentum at all.

Different strides for different directions

The second trick is tracking a separate stride for every value. Keep a record of how much each value has been swinging, then shrink the stride for ones that swing a lot and grow it for ones that barely move.

This helps most with a feature that rarely shows up. A value tied to something that appears only once in a while gets few chances to update, so growing its stride makes those rare chances count for more. Adam, a widely used optimizer, combines both tricks — momentum and per-direction strides — together.

The baseline stride is still a person's call

An optimizer adjusting the stride on its own doesn't make the learning rate unnecessary. What it adjusts is a multiplier; the baseline that multiplier gets applied to is still something a person sets. Set that baseline too high, and even good technique won't stop the bouncing.

That's why practice usually starts from a widely used optimizer as the default and tunes the learning rate first. Fixing the learning rate tends to shake up the outcome far more than swapping optimizers ever does.

3More precisely

An optimizer is a rule that takes the gradient plus a record of past updates and computes this step's update amount. The simplest form is plain stochastic gradient descent (SGD), which just multiplies the gradient by the learning rate. From there came a version that adds a momentum term carrying forward past movement, and one that divides the stride using the accumulated size of past gradients per value. The Adam family, which combines momentum with per-direction strides, is the most widely used today.

The comparison breaks down in places too. A relay track's curves and straightaways are visible, but training has no way to know in advance which direction is steep. It can only guess from how much wobble has shown up so far. And an optimizer keeps a separate record like this for every single value, which multiplies the memory a model needs during training many times over. A relay runner also only ever carries forward one baton's worth of speed, while an optimizer tracks a running history for every value independently, which is exactly why swapping in a new optimizer partway through a run tends to unsettle training for a while.

4Try it yourself

TensorFlow Playground ailearn.space Rerun training with nothing but the learning rate changed, and you'll get a feel for why a separate style of stepping is even needed

5Common misconceptions

It's easy to think switching to a better optimizer means the learning rate doesn't matter anymore, but actually the baseline step size is still a person's call, and that value swings the outcome more than anything else.
It's easy to think the optimizer finds a better answer, but actually it just descends the same landscape more efficiently — it can't change the landscape itself.
It's easy to think the newest optimizer is always best, but actually an older, simpler method often fits the data and model better, depending on the situation.

7One-line summary

In shortAn optimizer takes the downhill direction, carries forward its past speed, and gives each direction its own stride — it's the technique behind how the values actually move.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02