DQN,
explained.

Rung 1, so the comparison is against the Q-learning that preceded it. The algorithm was already understood and already provably convergent — keep a table of the expected return for every state-action pair, and repeatedly move each entry toward the reward you actually got plus the discounted best value available next. The catch is the table. For a game screen there are more states than could ever be enumerated, and no two are ever visited twice.

The change is to replace the table with a neural network taking raw pixels, which breaks the convergence argument in two specific ways worth naming because both fixes became universal. First, consecutive frames are almost identical, so training on experience in the order it arrives feeds the network a stream of highly correlated samples. Second, the network supplies its own target — the "best value available next" is its own output — so every update moves the target it is chasing.

The two stabilisers are the rung. Experience replay stores transitions in a large buffer and samples from it at random, which decorrelates the updates and lets each transition be learned from many times. A frozen target network is a periodically-updated copy used only to compute the target, so the target holds still between refreshes. The units to carry up: one architecture and one set of hyperparameters across roughly 49 Atari games, learning from pixels and score alone, reaching or beating a human on about half of them.

Reinforcement learning frames a task as: in some state s, pick an action a, get a reward r, land in a new state, repeat — and try to maximize total reward over time. A clean way to solve it is to learn the Q-function: Q(s, a) is the expected total future reward of taking action a in state s and playing well afterward. Know Q and the policy is trivial — in any state, take the action with the highest Q.

Classic Q-learning stores Q in a table, one entry per state-action. That's hopeless when the state is an Atari screen — millions of pixels, astronomically many states, none seen twice. DQN (Mnih et al., 2013/2015) replaces the table with a convolutional neural network that maps pixels straight to a Q-value for each action. Now the agent can generalize across similar screens instead of memorizing each one. The catch: training a neural network with Q-learning's bootstrapped targets was known to be unstable and prone to diverging. The paper's real contribution is the two ideas that fixed that — but first, the target it's chasing.

How do you train a value estimate when you never observe the "true" future reward directly? You bootstrap: build a target out of one real reward plus your own estimate of what comes next. That's the Bellman equation. The value of taking action a now should equal the immediate reward r plus the discounted value of acting optimally from the next state s':

The discount factor γ (near 1) makes distant rewards worth a bit less than immediate ones, keeping the sum finite and encoding "prefer reward sooner." The difference between the target and the current estimate is the temporal-difference (TD) error, and DQN's loss is simply the squared TD error — train the network to make each Q(s,a) match its Bellman target. The elegance is that you learn from single steps, correcting each guess with the very next reward, never needing to wait for an episode to end. The danger is equally clear: the target is built from the network's own estimates, so a network that drifts can feed itself bad targets and spiral. That's what the next two ideas prevent.

Naive neural Q-learning diverges for two reasons, and DQN adds one fix for each. First, experience replay: instead of training on each transition as it happens, store transitions (s, a, r, s') in a large buffer and train on random minibatches sampled from it. Consecutive frames of a game are almost identical — training on them in order feeds the network a stream of highly correlated, non-independent samples that skews the gradient. Random sampling from the buffer breaks that correlation (restoring the i.i.d. assumption gradient descent wants) and lets each experience be reused many times, which is far more data-efficient.

Second, the target network: DQN keeps a second, frozen copy of the Q-network and uses it to compute the Bellman target y, updating this copy to match the online network only periodically (every 10,000 parameter updates in the paper). Why? If you compute the target with the same network you're updating, the target shifts the instant you take a gradient step — you're chasing a target that runs away from you, a feedback loop that easily diverges. Freezing the target for a stretch gives the online network a stationary goal to move toward, then updates the goal occasionally. These two tricks — decorrelated data and a stable target — are the difference between neural Q-learning that blows up and one that learns Atari. Neither is deep math; both are engineering insight about what makes the optimization well-posed.

The Bellman target for a transition, with reward r, discount γ, and next-state values from the target network:

r=1, γ=0.9, and next-state Q-values [2, 5, 3] → y = 1 + 0.9·5 = 5.5. On a terminal transition the future term vanishes and y = 1. The squared gap to the current Q is the loss.

Learning is a step of the current estimate toward the target, by the learning rate α — the tabular view of the same gradient update:

From Q(s,a)=3 with target 5.5 and α=0.1: Q ← 3 + 0.1·2.5 = 3.25 — a small move toward the target. The greedy policy at test time just takes the arg max. The runnable version below computes the target, the TD error, the update, and the greedy action.

The agent sees only screen pixels and the game score. All 49 games use the same network design and the same settings; only the number of output actions changes from game to game.

1. Preprocess each frame. Take the pixelwise maximum of the current and previous frame (some sprites only draw on alternate frames), convert color to brightness, and shrink the image to 84×84. Stack the last four processed frames into one 84×84×4 input. A single frame cannot show which way a ball is moving; four can.
2. Repeat each action (frame-skip). The agent chooses an action only on every fourth frame and that action is repeated on the frames in between. That lets it play about four times as many steps for the same compute. One game, Space Invaders, uses three instead, because at four the lasers blink out of sight.
3. Read all Q-values in one pass. The network has three convolutional layers, one fully connected layer of 512 units, and one output per legal action (between 4 and 18 depending on the game). One forward pass scores every action, which is cheaper than running the network once per action.
4. Pick the action with ε-greedy. With probability ε take a random action; otherwise take the highest-scoring one. ε starts at 1.0, falls linearly to 0.1 over the first million frames, and then stays at 0.1. At evaluation time it is 0.05. Before any learning, about 50,000 frames of purely random play fill the memory.
5. Store the transition. Each step writes (state, action, reward, next state) into a replay memory that keeps the most recent million transitions and drops the oldest.
6. Learn from a random minibatch. Every fourth action, sample 32 transitions uniformly from the memory, compute a target for each with the frozen copy of the network, and take one RMSProp step on the online network. Rewards are clipped to −1, 0 or +1, and the error term is clipped to the range −1 to +1 so no single step produces a huge gradient. The discount is 0.99.
7. Refresh the target copy. Every 10,000 parameter updates, copy the online weights into the target network. Between copies the target network does not move.
8. Repeat for 50 million frames, roughly 38 days of game time per game.

Why off-policy learning permits replay

Transitions in the memory were produced by older versions of the network, with different ε. That is only usable because Q-learning is off-policy: its target takes the best next action according to the current estimate, whatever action the behavior policy actually took. The data may come from one policy while the quantity being learned describes another. A method that estimates the value of the policy that gathered the data would be learning from stale data about a policy that no longer exists, which is the problem the next paper has to design around.

What the clipping trades away

Game scores run from single digits to thousands. Clipping rewards to their sign keeps the error scale similar across games, so one learning rate works for all of them. The cost is that the agent cannot tell a small reward from a large one. Clipping the error term makes the loss behave like squared error for small mistakes and like absolute error for big ones.

• θ — weights of the online network, the only ones trained by gradient steps.
• θ^− — the frozen copy used for the target, overwritten with θ every 10,000 updates.
• U(D) — uniform sampling from the replay memory D.
• r — the clipped reward; on a terminal step the target is just r.
• γ — the discount, 0.99.

Worked example: r = 1, γ = 0.99 and target-network values [2, 5, 3] for the next state give y = 1 + 0.99 × 5 = 5.95. If the online network says Q(s, a) = 3, the error is 2.95, but the clipped error used for the gradient is 1.

Zero means a random player and 100 means the professional human tester used as the baseline. This is how results on games with very different score ranges get averaged.

The setup was 49 Atari 2600 games, 50 million frames of training per game, evaluation with ε = 0.05, and a comparison against a professional human tester and against the best earlier linear-feature learner. The agent scored above 75 percent of the human level on 29 of the 49 games, and beat the best earlier learner on 43 of them. It did very badly on games that need long, deliberate exploration; Montezuma's Revenge is the standing example.

The ablation removed the two stabilizers one at a time on a handful of games. On Breakout the full agent averaged about 317 points. Without replay, but with the target network, it fell to about 241. Without the target network, but with replay, it fell to about 10. Without either it scored about 3. Other games in the ablation moved in the same direction, though by smaller factors.

The pattern is the useful part. Both tricks matter, and neither is a small tuning detail: the same network and loss with either switched off is close to useless on this game. The one-recipe result also stands: a single set of hyperparameters, chosen on a few games, carried across all 49 without per-game changes.

The reported failures are specific and were not hidden: the method does badly on games where reward is far from the action that earned it and exploration has to be deliberate rather than random — Montezuma's Revenge is the standing example, and it became a benchmark for exploration research on the strength of this paper naming it. The sample cost is also enormous, tens of millions of frames per game, which is many human lifetimes of play.

One defect was found rather than admitted. Taking a maximum over noisy value estimates systematically overestimates, because the max of several noisy numbers is biased upward — van Hasselt and colleagues showed this is not a small effect and fixed it by using one network to choose the action and another to value it. Double DQN, duelling architectures and prioritised replay followed in quick succession, which is the sign of a result that was correct enough to be worth improving.

How it held up: as an algorithm it is largely superseded; as infrastructure it is everywhere. Replay buffers are standard in every off-policy method, and a slowly-updated copy of a network used as a stationary target reappears far outside reinforcement learning — the momentum teacher on the vision ladder is the same device with a different purpose. Its structural limit is that it learns a value for each discrete action, so it cannot act in a continuous space at all, which is where rung 2 starts.

DQN is the paper that kicked off deep reinforcement learning as a field. Before it, RL and deep learning were largely separate; after it, "a neural network learning to act from high-dimensional input by reward" became a research program that produced a decade of milestones. The direct line runs to AlphaGo and its successors, and to the value- and policy-learning ideas that underpin how we train agents today.

Its two engineering fixes also outlived their original setting. Replay buffers and target (or slowly-updated "Polyak-averaged") networks became standard components in off-policy RL far beyond DQN, and the general lesson — that bootstrapped self-referential targets need stabilization to be trainable — recurs whenever a model is trained against a target derived from itself. DQN handles discrete actions with a value function; the field's other main branch, policy-gradient methods like PPO, tackles continuous control and now the fine-tuning of language models. Between them, value-based and policy-based, sits nearly all of modern RL — and DQN is where the "deep" in it began.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Reward, policy, search. The line from Atari to a Go engine that was never shown a human game — and the algorithm (PPO) that RLHF later borrowed wholesale.

1. 1DQN2015
2. 2PPO2017
3. 3AlphaGo2016
4. 4AlphaZero2018
5. 5MuZero2020

All 110 papers, in 19 ladders →