Gradient Descent

What goes down is "how wrong"

In the garage, height goes down. In training, error goes down — a single number for how far the model's answer strayed from the correct one. A big number means you're still up top; a small one means you've come a long way down.

The dials that move this number sit inside the model: the values adjusted during training, the weights and biases attached to every connection. Turn one dial and the error nudges up or down a bit. Gradient descent is the act of turning all these dials, a little at a time, in whichever direction lowers the error.

A large model has anywhere from millions to hundreds of billions of these dials. That's like standing on a floor that can tilt in millions of directions at once — impossible to draw, but the math works the same no matter how many directions there are.

The slope is read from right underfoot

The model can't survey the whole landscape. All it can see is the slope at the exact spot it's standing on. Nudge one dial by a hair and check how much the error rises or falls; do that for every dial, and the whole bundle becomes the downhill direction from here.

The procedure that actually runs this calculation is backpropagation. Once an answer comes out, it takes the error and works backward through the layers, handing each dial its own share: "you, move this much this way." Gradient descent takes that direction and moves the values accordingly.

Seeing only what's underfoot is both a weakness and a strength. You'll never know for certain where the lowest point is far away. But you also never have to search the whole landscape, which is exactly what lets a single step get computed even with hundreds of billions of dials. Testing every possible spot one by one would mean training never even starts.

Step size and style are handled separately

Knowing the downhill direction doesn't settle how far to go. How much to move in one step is set separately, by the learning rate. Set it big and you descend fast, but you might sail past the bottom and climb the opposite wall; set it small and it's safe, but you can spend all day on the same floor.

The style of taking that step belongs to the optimizer. It speeds things up when the direction stays consistent, and it can size the step differently depending on the direction. Direction comes from gradient descent, size comes from the learning rate, and style comes from the optimizer.

Flat doesn't always mean bottom

Keep descending and eventually you hit a spot surrounded by uphill on every side. There's nowhere left to go down, so the steps stop. But that spot isn't guaranteed to be the garage's lowest floor — it could just be a shallow dip in the middle of some random level.

Training still tends to work out, because with enough dials, a spot that's uphill in every single direction at once turns out to be rare, and most stopping points are low enough to be useful. Splitting the data into small chunks helps too — the slope underfoot shifts slightly each time, which helps shake the model loose from a shallow dip.