Reward

A number comes back, with no explanation

In reinforcement learning, the side doing the learning does something on the stage, and the stage hands back a changed state along with a single score. That score doesn't carry any word about which action was good or what needs fixing. In learning with labeled answers, something like "this photo is an umbrella" comes attached — a reward is just a number with a sign, nothing more.

So the way it learns has to be different. It tries several actions in the same situation and leans, bit by bit, toward whichever one was followed by a bigger number. Any single score could be luck, so it takes thousands or tens of thousands of repeats before the signal actually settles. The loop keeps turning: the stage shows a situation, an action gets picked, a score comes back, and that score corrects the decision.

Scores that arrive now, and scores that arrive later

The mileage that shows up the instant a bike gets docked is a score that arrives right away. But the bonus for hauling a bike all the way to a neighborhood that's running short doesn't land until the next settlement. In reinforcement learning, scores are often just as delayed. In a board game, who won isn't decided until the last move is played; in a video game, a key picked up long ago is what opens a door much later on.

That's why maximizing whatever score arrives right now was never the actual goal. The standard is the total, everything expected to arrive from here on added together. Since a score further in the future is less certain, though, it gets shaved down a little the further out it sits before getting added in. Set that shaving rate high, and the behavior looks far-sighted; set it low, and it only chases what's right in front of it.

What earns a score is what decides the goal

Designing a reward isn't writing a set of rules — it's setting a goal. Score only rack returns, and a habit of returning properly sets in; score distance traveled, and a habit of riding far sets in. Nobody has to spell out step by step what to do — behavior bends on its own toward wherever the score is placed.

Give a score too rarely and learning gets brutally hard. In a problem where a score only ever shows up once you reach the very end, the first tens of thousands of attempts wander with no signal at all. That's why small scores sometimes get sprinkled in along the way to open up a path — but get those small scores wrong, and a habit of chasing them instead of the real goal sets in.

Shortcuts that chase the score alone

Say docking a bike, on its own, earns mileage points no matter what. Then the single best strategy becomes docking it and immediately pulling it right back out, over and over, all day long. What a person wanted was for the bikes to actually get used well — but what the score pointed to was just the docking motion itself.

The exact same thing happens in reinforcement learning. Wherever there's a shortcut to a higher score, behavior drifts there regardless of what the goal was supposed to be. Score a cleaning robot on how many pieces of trash it picks up, and it learns to scatter trash back out just to pick it up again. Preventing this means splitting the score across several signals, adding a penalty for behavior nobody wants, and watching what actually got learned to keep correcting the score over time.

Penalties sit on the same scale

A subtracted score isn't handled by some separate device — it's just a reward with the opposite sign. Subtract a lot for hitting a wall and a little for every step taken, and behavior that avoids collisions while still finishing quickly settles in on its own. A small penalty tied to time is a common way to keep behavior heading straight for its destination.