DQNDeep Q-Network
A neural network standing in for a table of expected scores
- DQN (Deep Q-Network) is a method that has a neural network remember the expected score for every action, instead of a table.
- Feed it a situation and it hands back the value of every action at once. Even for a situation it has never once faced, it guesses a value based on similar ones it has.
- What it has experienced gets stored away and pulled back out at random, shuffled, for further training — a device that stops it from tilting toward whatever it just happened to see in a row.
- The value it aims for is pulled from a copy frozen in place for a while, since letting the target wobble every single step throws training into chaos.
- Learning to play a game straight from raw screen pixels first caught on widely with this method, and it went on to become the basic skeleton of reinforcement learning ever since.
Contents
1The analogy
Imagine pricing every seat in a thirty-thousand-seat concert venue. Printing a numbered slip for each individual seat would take thirty thousand slips, and a brand-new venue would mean starting the whole thing over from scratch. So in practice, nobody does it that way. Instead, a formula gets built that spits out a value once you feed in a few things — which level, distance from the stage, how much the view gets blocked. The table disappears; only the formula is left. That way, even a seat that's never been priced before gets a value right away, and neighboring seats naturally get similar values. Watch how tickets actually sell and nudge the formula a little, and the value of every single seat improves all at once. Building this formula is exactly the job DQN does.
2In detail
A machine that hands back a value instead of writing down a square
The table approach writes down a separate number for every place where one situation meets one action. Once situations get as complex as raw screen images, the square count balloons past what anything could handle, and most squares are left never filled in at all.
DQN swaps this table out for a neural network. Feed in the current situation, and the value of every available action comes back at once, in a single row. Four possible actions means four numbers arrive together, and whichever is biggest becomes the action taken. Correcting a value doesn't mean erasing one square and rewriting it either — it means nudging the values inside the network a little, so its output for that situation edges closer to the target.
What's learned in one situation carries over to similar ones
The table approach's biggest cost was that every square stood alone. However similar a neighboring square's situation might be, nothing learned there carried over. A neural network works the opposite way. Situations that look alike produce values that come out alike. What's learned in one spot spreads out across the whole neighborhood around it.
That's what lets it produce a plausible value even for a situation it's never once faced. It's the reason a reasonable action can still get picked in front of a screen it has never seen before. Of course, since it's a guess, it can also be wrong — the more unfamiliar the situation, the more the value can come out way off.
What's been experienced gets stored, then shuffled back out
The scenes encountered over the course of one round sit right up against each other in time. What just appeared on screen and what appears next are nearly identical. Feed scenes like that into training in order, and the network ends up fitting only to the last few seconds and forgetting whatever it learned earlier.
So whatever gets experienced is stored away in memory instead — what the situation was, what got picked, what score came back, and what situation it led to, bundled together as one unit. During training, a random handful gets pulled out of that memory. Mixing scenes from different points in time cuts down the tilt toward any one stretch, and reusing an experience several times over also makes things more efficient.
Why the target gets held still for a while
When correcting a value, the number aimed for is the biggest value at the next situation. But that number comes from the very same network doing the correcting. When the thing being corrected and the target it's aiming for live in the same body, nudging one drags the other along with it — like measuring a length with the same ruler you're pushing around. The values can swing out of control.
So a copy gets made, and the target value is pulled only from that copy. The copy stays frozen for a while and only gets refreshed to the latest state now and then. Holding the target still for a while is what settles the training down. Shuffling experience back out of storage and keeping this frozen copy are the two devices that made DQN actually workable.
What actually changed
The skeleton is straight out of Q-learning. The score just received, plus the biggest value at the next situation shaved down a little, becomes the target, and the current value gets nudged toward it a bit at a time. What changed is where the value gets stored. Moving it from a table to a neural network is what broke the limit on how many situations there could be.
New problems came along with it, though. A value the network produces is a guess, prone to running higher than reality, and once training drifts off course it doesn't always find its way back. Fixes followed close behind: splitting the job of picking an action from measuring it, learning the value of the situation separately from the difference each action makes, and pulling more often from whichever stored experience has more to teach.
3More precisely
DQN approximates the action-value function with a neural network, and stabilizes training by pairing it with an experience replay buffer and a target network. The loss is the gap between the target value and the current output, minimized through gradient descent, where the target value is the reward received plus a discounted version of the best next-state value produced by the target network. When raw pixels are the input, a stack of convolutional layers sits up front, and several recent frames get layered together so motion is visible. Since the action has to come from a countable set of options, a continuous action like a steering angle calls for a different family of methods.
The analogy breaks down in a few places. A seat-pricing formula is built by a person choosing which factors matter, but DQN decides on its own what even matters to look at — it works out which pixel patterns matter without ever being told. And a seat's price has a solid answer in actual ticket sales, while what DQN is chasing is just another guess produced by its own frozen copy. Chasing a guess with a guess means that when things go wrong, the whole set of values can drift off course together.
4Try it yourself
- Balance the Pole ailearn.space Watch a neural network learn to keep a pole from falling over. See the early stumbling get ironed out over just a few hundred rounds, right from the start
- DQN Car ailearn.space Feed in the screen state and out come values for each control, and the biggest one wins. Toggle training on and off and compare how the values change
5Common misconceptions
It's easy to think DQN is a new theory that replaced Q-learning, but actually it keeps the same update rule and only swaps out where the value gets stored, from a table to a neural network.
It's easy to think a neural network always beats the table approach, but actually with only a few dozen possible situations, a table is faster and steadier.
It's easy to think DQN works for any kind of action, but actually it only works when the actions available form a countable set.
7One-line summary
In shortDQN swaps the table that used to hold an expected score for every action with a neural network, letting it guess a value even in situations it has never faced before.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02