DQN,
explained.

Reinforcement learning frames a task as: in some state s, pick an action a, get a reward r, land in a new state, repeat — and try to maximize total reward over time. A clean way to solve it is to learn the Q-function: Q(s, a) is the expected total future reward of taking action a in state s and playing well afterward. Know Q and the policy is trivial — in any state, take the action with the highest Q.

Classic Q-learning stores Q in a table, one entry per state-action. That's hopeless when the state is an Atari screen — millions of pixels, astronomically many states, none seen twice. DQN (Mnih et al., 2013/2015) replaces the table with a convolutional neural network that maps pixels straight to a Q-value for each action. Now the agent can generalize across similar screens instead of memorizing each one. The catch: training a neural network with Q-learning's bootstrapped targets was known to be unstable and prone to diverging. The paper's real contribution is the two ideas that fixed that — but first, the target it's chasing.

How do you train a value estimate when you never observe the "true" future reward directly? You bootstrap: build a target out of one real reward plus your own estimate of what comes next. That's the Bellman equation. The value of taking action a now should equal the immediate reward r plus the discounted value of acting optimally from the next state s':

The discount factor γ (near 1) makes distant rewards worth a bit less than immediate ones, keeping the sum finite and encoding "prefer reward sooner." The difference between the target and the current estimate is the temporal-difference (TD) error, and DQN's loss is simply the squared TD error — train the network to make each Q(s,a) match its Bellman target. The elegance is that you learn from single steps, correcting each guess with the very next reward, never needing to wait for an episode to end. The danger is equally clear: the target is built from the network's own estimates, so a network that drifts can feed itself bad targets and spiral. That's what the next two ideas prevent.

Naive neural Q-learning diverges for two reasons, and DQN adds one fix for each. First, experience replay: instead of training on each transition as it happens, store transitions (s, a, r, s') in a large buffer and train on random minibatches sampled from it. Consecutive frames of a game are almost identical — training on them in order feeds the network a stream of highly correlated, non-independent samples that skews the gradient. Random sampling from the buffer breaks that correlation (restoring the i.i.d. assumption gradient descent wants) and lets each experience be reused many times, which is far more data-efficient.

Second, the target network: DQN keeps a second, frozen copy of the Q-network and uses it to compute the Bellman target y, updating this copy to match the online network only every few thousand steps. Why? If you compute the target with the same network you're updating, the target shifts the instant you take a gradient step — you're chasing a target that runs away from you, a feedback loop that easily diverges. Freezing the target for a stretch gives the online network a stationary goal to move toward, then updates the goal occasionally. These two tricks — decorrelated data and a stable target — are the difference between neural Q-learning that blows up and one that learns Atari. Neither is deep math; both are engineering insight about what makes the optimization well-posed.

The Bellman target for a transition, with reward r, discount γ, and next-state values from the target network:

r=1, γ=0.9, and next-state Q-values [2, 5, 3] → y = 1 + 0.9·5 = 5.5. On a terminal transition the future term vanishes and y = 1. The squared gap to the current Q is the loss.

Learning is a step of the current estimate toward the target, by the learning rate α — the tabular view of the same gradient update:

From Q(s,a)=3 with target 5.5 and α=0.1: Q ← 3 + 0.1·2.5 = 3.25 — a small move toward the target. The greedy policy at test time just takes the arg max. The runnable version below computes the target, the TD error, the update, and the greedy action.

DQN was the proof that neural networks and reinforcement learning could combine to learn control from raw perception:

DQN wasn't flawless — it overestimates values (later fixed by Double DQN), it's sample-hungry, and it only handles discrete actions. A wave of refinements followed (Double DQN, Dueling networks, prioritized replay, Rainbow that combined them). But the core demonstration stood: reward alone, plus pixels, plus a stabilized bootstrapped value function, is enough to learn skilled behavior end to end.

DQN is the paper that kicked off deep reinforcement learning as a field. Before it, RL and deep learning were largely separate; after it, "a neural network learning to act from high-dimensional input by reward" became a research program that produced a decade of milestones. The direct line runs to AlphaGo and its successors, and to the value- and policy-learning ideas that underpin how we train agents today.

Its two engineering fixes also outlived their original setting. Replay buffers and target (or slowly-updated "Polyak-averaged") networks became standard components in off-policy RL far beyond DQN, and the general lesson — that bootstrapped self-referential targets need stabilization to be trainable — recurs whenever a model is trained against a target derived from itself. DQN handles discrete actions with a value function; the field's other main branch, policy-gradient methods like PPO, tackles continuous control and now the fine-tuning of language models. Between them, value-based and policy-based, sits nearly all of modern RL — and DQN is where the "deep" in it began.