Learn from
nothing.

This rung is a subtraction. AlphaGo needed a corpus of expert human games to bootstrap its policy, a hand-designed fast rollout policy to score simulations, and several separate networks. AlphaZero removes all three and is stronger. One network with two heads — a move distribution and a single value — is trained from random initialisation by self-play alone, and the search evaluates leaves with the value head rather than by playing the game out.

What replaces the human data is a loop that is worth stating precisely, because it is the most reusable idea on the ladder. The search is a policy improvement operator. Run tree search from a position and the visit counts over moves are a better move distribution than the raw network prior that guided the search — the search has done work the network had not. So train the network's policy head toward the search's visit counts, and its value head toward the eventual game result. A better network then makes the next search better, and there is no ceiling supplied by human play because no human play is involved.

Stated in units: the same algorithm, with the same hyperparameters, reaches superhuman strength at Go, chess and shogi — three games whose previous champion programs shared no code and, in chess's case, embodied decades of hand-encoded domain knowledge. The generality is the result; the individual game strengths are the evidence for it.

Game-playing AI had always leaned on human knowledge: opening books, handcrafted evaluation functions, decades of grandmaster games. AlphaGo beat the world champion at Go but still bootstrapped from human games and Go-specific features. AlphaZero (Silver et al., 2017) stripped all of that away. It's given only the rules — no human data, no opening theory, no domain heuristics — and learns entirely by self-play: the current network plays games against itself, and the outcomes make it better. The same algorithm, unchanged, mastered Go, chess, and shogi.

The engine is a marriage of two things neural networks and classical AI each do well. A single deep network looks at a position and outputs fast intuition: a policy (which moves look good) and a value (who's likely to win). A Monte Carlo Tree Search then uses that intuition to look ahead deliberately, exploring lines of play. Search produces better moves than the raw network; those better moves train the network; the improved network makes search better still. Round and round, from nothing to superhuman.

The network f(s) = (p, v) maps a board state s to a policy vector p (a probability for each legal move) and a scalar value v ∈ [−1, 1] (expected outcome from this position). That's the intuition: instant, but shallow — it hasn't looked ahead. MCTS supplies the depth by building a search tree, one simulation at a time:

A key simplification over older Monte Carlo methods: AlphaZero doesn't play out random games to the end to estimate a position. The network's value head is the evaluation, and the policy head focuses the search on plausible moves, so far fewer simulations are needed. After running many simulations from the root, the visit counts over the root's moves form an improved policy π — sharper than the network's raw p, because it reflects look-ahead. That improved policy is both what AlphaZero plays and what it trains toward.

The heart of the search is how it decides which move to explore next at each node. AlphaZero uses PUCT (Predictor + Upper Confidence bounds for Trees): pick the action maximizing an exploitation term plus an exploration bonus.

Q = mean value seen for move a (exploit what's working). P = the network's policy prior (steer toward good-looking moves). N = visit counts; the √(parent visits)/(1+child visits) shape makes unvisited moves attractive early, then fades. c_puct sets the exploration strength.

Read the two terms. Q(s,a) is exploitation — the average result this move has produced so far, so proven-good moves rise. U(s,a) is exploration — large when a move has a high policy prior P but few visits N, and it shrinks as visits accumulate. Early in the search, unvisited moves with strong priors get tried; as evidence piles up, the 1 + N denominator damps the bonus and attention concentrates on whatever actually has the best Q. The network's prior P steering the bonus is what makes the whole search tractable in games with enormous branching factors — you don't waste simulations on moves the network considers absurd.

Each backup updates a node's value as a running mean of the outcomes seen through it:

A new evaluation v is folded into the existing average Q over N prior visits. Over many simulations, Q converges to the search's estimate of the move's value.

And the network is trained to match what search and self-play reveal — the improved policy π (from visit counts) and the game's actual outcome z ∈ {−1, 0, +1}:

The value head is regressed toward the real outcome z; the policy head is pushed (cross-entropy) toward the search-improved policy π; a weight-decay term regularizes. One network, two heads, one combined loss.

That's the whole system: self-play generates games; MCTS with PUCT turns the network's rough intuition into strong moves and a sharpened policy; the network trains on those improved policies and final outcomes; the better network makes the next round of search stronger. The runnable version below implements the PUCT score, the move selection, and the value backup — the decision-making core of the search.

This is the full recipe once the search rule and the two-headed network are taken as given. The point of the paper is how little else there is: one network, one loop, and no game-specific tuning beyond the board encoding.

1. Encode the position as stacked planes. The input is a stack of binary and numeric planes covering the last several positions plus extras such as castling rights and repetition counts for chess. For chess that comes to 119 planes on an 8×8 board; for shogi it is 362 planes on 9×9. Nothing else about chess or shogi strategy is supplied.
2. Encode moves as planes too. The policy head outputs a probability for every (square, move type) pair. Chess uses 8×8×73, which is 4,672 possible moves. Shogi uses 9×9×139, which is 11,259. Illegal moves are masked out before the probabilities are normalized.
3. Start from random weights. The first network plays essentially random games. There is no opening book, no human game, and no evaluation function.
4. Search 800 simulations per move. At each turn of self-play, run 800 tree-search simulations guided by the current network. During self-play, the root prior gets Dirichlet noise whose concentration is scaled inversely to the typical number of legal moves: 0.3 for chess, 0.15 for shogi and 0.03 for Go.
5. Pick the move from visit counts. Early in a game, sample a move in proportion to visit counts (temperature 1), which keeps the openings varied. Later, take the most-visited move. The exact cutoff is a training detail this page does not pin down.
6. Record (position, π, z). Store each position with the visit-count distribution π from the search. When the game ends, fill in the outcome z for every stored position: +1 for a win, 0 for a draw, −1 for a loss, from that position's player's point of view.
7. Train continuously on recent games. Sample minibatches of 4,096 positions from a pool of recent self-play games and update the single network. The run used 700,000 such steps. Self-play ran on 5,000 first-generation TPUs while 64 second-generation TPUs did the training.
8. Always play with the newest network. There is no tournament before a new network is allowed to generate games.

What changed from AlphaGo Zero

AlphaGo Zero predicted the probability of winning, treated the outcome as win or loss, and used the symmetry of the Go board in two ways: it augmented training data with rotations and reflections, and it evaluated search positions under randomly chosen rotations and reflections. AlphaZero estimates an expected outcome, so draws are first-class, and uses no symmetry because chess and shogi boards are not symmetric. AlphaGo Zero also kept a best-player checkpoint and replaced it only when a challenger won 55 percent of a 400-game match; AlphaZero drops that gate. Finally, AlphaGo Zero's settings were tuned for Go; AlphaZero reuses one set for all three games.

• z — the game outcome, +1, 0 or −1; allowing 0 is the draw-aware change from AlphaGo Zero.
• v — the value head's prediction; trained toward z by squared error.
• π — the search's visit-count distribution, the target for the policy head.
• p — the network's raw move probabilities; the cross-entropy term pulls p toward π.
• c ‖θ‖² — weight regularization.
• τ — temperature; 1 early in the game, later close to 0, which means pick the most-visited move.

Worked example: 800 simulations split 600, 150 and 50 over three moves give π = (0.75, 0.1875, 0.0625). If the network had p = (0.5, 0.3, 0.2), the policy term is −(0.75 ln 0.5 + 0.1875 ln 0.3 + 0.0625 ln 0.2) ≈ 0.85. With a drawn game (z = 0) and v = 0.4, the value term is 0.16.

Each game was trained from scratch with the same algorithm and the same settings. Reported milestones: in chess, AlphaZero passed Stockfish after about 4 hours (around 300,000 training steps); in shogi it passed Elmo after about 2 hours (around 110,000 steps); in Go it passed the AlphaGo version that beat Lee Sedol after about 8 hours (around 165,000 steps).

The search efficiency comparison is the surprising part. In chess, AlphaZero examined about 80,000 positions per second, against about 70 million for Stockfish; in shogi about 40,000 against about 35 million for Elmo. A learned prior and value let it search roughly a thousand times fewer positions and still win. When the thinking time per move was varied, its tree search scaled at least as well as the conventional engines' alpha-beta search.

The comparison was criticized afterward: Stockfish ran with a fixed hash size and a fixed per-move time, without an opening book, and on hardware not directly comparable to AlphaZero's TPUs. The outcome was not seriously disputed, but the exact margin depends on those choices.

The requirement the paper does not remove, and is explicit about, is the one that bounds the whole approach: it must be given a perfect model of the rules. Self-play needs to know what a legal move is, what state it leads to, and when the game is over. That is available for board games and almost nothing else, which names rung 5's job exactly.

The headline comparisons were contested on methodology rather than outcome. The chess result against a strong conventional engine drew criticism over hardware asymmetry, time controls, opening books and the engine's configuration; the substance of the achievement was not seriously disputed, and later open reproductions supported it. Worth separating: "this works" was never in question, "by exactly this margin against that opponent" was.

How it held up: as the canonical demonstration that learned intuition plus search beats either alone, and as the origin of the loop above. Its influence on language models is indirect but real — training a model on the output of a more expensive procedure that used the model itself is what bootstrapped reasoning does with generated solutions instead of tree search, and the reason both work is the same: the expensive procedure produces a signal the cheap model did not already contain.

AlphaZero distilled game-playing AI into a clean, reusable idea: pair a network that supplies learned intuition with a search that supplies deliberate lookahead, and let self-play improve both. Its direct descendant MuZero removed the last piece of hand-given knowledge — the rules — by learning a model of the environment's dynamics and planning inside it, extending the recipe to Atari and settings where the simulator isn't known.

The influence reaches well past board games. The "combine fast learned policies with slow deliberate search" pattern echoes across reinforcement learning — it complements value-based methods like DQN and policy-gradient methods like PPO by adding explicit planning on top of learned evaluation. And the deeper theme — that test-time search over a learned model can unlock capabilities the base model lacks on its own — resurfaced in modern language models, where letting a model deliberate, branch, and evaluate its own reasoning at inference (rather than answering in one shot) mirrors exactly what MCTS does over a game tree. AlphaZero's marriage of intuition and search remains one of AI's most quietly generative ideas.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Reward, policy, search. The line from Atari to a Go engine that was never shown a human game — and the algorithm (PPO) that RLHF later borrowed wholesale.

1. 1DQN2015
2. 2PPO2017
3. 3AlphaGo2016
4. 4AlphaZero2018
5. 5MuZero2020

All 110 papers, in 19 ladders →