AlphaGo,
explained.

Rungs 1 and 2 are both reactive: a state goes in, an action comes out, and all the computation that improved the decision happened during training. Neither spends extra effort at decision time on a hard position. AlphaGo's change is to put a search between the network and the move, so the quality of a decision can be bought with compute at the moment it is needed.

Why Go forced this: the branching factor is on the order of 250 legal moves per position against roughly 35 for chess, so searching to any useful depth by enumeration is hopeless, and unlike chess there is no good hand-written function for judging who is ahead. Both of the things that made computer chess work were unavailable.

The construction supplies both from learning, and the division of labour is the thing to carry up the ladder. A policy network proposes a short list of plausible moves, which prunes the 250-wide tree to something searchable. A value network estimates who is winning from a position, which removes the need to search to the end of the game. Monte Carlo tree search then spends its budget where it matters, selecting branches by a rule that balances the policy's prior and the estimated value against how often a branch has already been tried — exploit what looks good, but not so exclusively that a good move with a low prior is never examined. The networks were bootstrapped from a large corpus of human expert games and then improved by self-play.

Chess programs win by searching: look ahead many moves, evaluate the resulting positions with a hand-crafted scoring function, and pick the line that leads somewhere good. Go breaks this on two axes. Its branching factor is enormous — around 250 legal moves per turn versus chess's ~35 — so the tree explodes far faster. And no one knew how to write a good evaluation function for a Go position; the value of a board is famously subtle, resisting the tidy piece-counting that works in chess.

So AlphaGo's problem was really two problems: the tree is too wide to search fully, and positions are too subtle to score by rule. Its answer was to learn both from data. A policy network learned which moves are worth considering — narrowing the tree — and a value network learned to score positions — replacing the missing evaluation function. Deep learning supplied the intuition; tree search supplied the calculation. Neither alone was enough; together they were superhuman.

Two deep networks read the board. The policy network outputs a probability over moves — a prior saying which moves look good before any search. This is what tames the branching factor: instead of considering all ~250 moves seriously, the search concentrates on the handful the policy rates highly. The value network outputs a single number — an estimate of the probability that the current player wins from this position. This is the learned evaluation function Go never had.

Both are crucial to making search tractable. Earlier Go programs evaluated a position by playing it out to the end with fast random moves (Monte Carlo rollouts) — noisy and slow. AlphaGo's value network estimates the outcome directly from the position, and a separate small, fast rollout policy handles whatever rollouts it still does. The networks turn an impossible search into a guided one.

Monte Carlo Tree Search grows a search tree one simulation at a time. Each simulation: select a path down the tree, expand a new leaf, evaluate it with the value network, and back up the result to update every node on the path. The heart is the selection rule, PUCT, which decides at each node which move to explore by balancing exploitation against exploration:

(Strictly, the √ΣN factor is the PUCT form introduced with AlphaGo Zero; the original AlphaGo bonus was proportional to P(a)/(1 + N(a)).) The first term Q(a) = W/N is the move's average result so far — pure exploitation, pick what's been working. The second rewards moves with high policy prior P(a) and few visits N(a) — exploration, try promising or under-explored moves. Early on, the prior dominates and search follows the policy network's intuition; as a move accumulates visits, its exploration bonus shrinks and the choice increasingly trusts its measured value. Each simulation backs up its leaf value, incrementing visit counts and refining Q. After thousands of simulations, the most-visited move at the root is played. The runnable version below shows PUCT favoring the prior early and the value late.

AlphaGo's networks were trained in stages, each building on the last:

Starting from human games gave the policy a strong initialization; self-play then pushed it past human play by discovering moves no human database contained. The value network, trained on the outcomes of these self-play games, learned to judge positions the way a strong player would. Its successor, AlphaGo Zero, later removed the human data entirely — learning from pure self-play, tabula rasa, and surpassing the original.

AlphaGo is not one network. It is a training pipeline that produces four learned components, and a search that uses all of them at play time.

1. Train a supervised (SL) policy network on human games. A 13-layer convolutional network reads the board and predicts the move an expert played. It was trained on about 30 million positions from a public online Go server and predicted the expert move about 57 percent of the time, against about 44 percent for the best earlier result.
2. Train a small, fast rollout policy. A linear model over simple local board patterns predicts expert moves about 24 percent of the time, but picks a move in about 2 microseconds, against about 3 milliseconds for the big network. It exists only to play simulated games to the end quickly.
3. Improve a copy by self-play (the RL policy). Start from the SL network, then play it against randomly chosen earlier versions of itself, rewarding +1 for a win and −1 for a loss with a policy-gradient update. Opponents are drawn from older versions so the policy does not overfit to one rival. This network beat the SL network about 80 percent of the time head to head.
4. Train a value network on self-play outcomes. A network with the same body as the policy network but a single output regresses toward who won. Training on every position of a game overfit badly, because positions from the same game share one outcome and the network memorized them. So the training set holds one position from each of 30 million distinct self-play games.
5. Select inside the tree. Every edge stores a prior P (from the SL policy), a visit count N and an average value Q. From the root, walk down by taking the move with the highest Q plus a bonus that is large for high-prior, rarely visited moves and shrinks as N grows.
6. Expand and evaluate the leaf two ways. At a new leaf, the policy network supplies priors for its children, then the leaf is scored twice. The value network gives one estimate from a single forward pass. The fast rollout policy plays the game to the end and yields a win or a loss. The two are mixed as V = (1 − λ)v + λz, with λ = 0.5 working best of the values the authors tried.
7. Back up. Each edge on the path updates its visit count and sets Q to the average of the leaf evaluations that passed through it.
8. Play the most-visited move at the root. Visit count is more stable than raw value, so it is the choice rule.

Why the prior comes from the SL policy

The RL policy was the stronger player on its own, but inside the search the SL policy worked better as the prior. The authors' explanation is that human moves are a broader set of reasonable candidates, while the RL policy concentrates on the single move it prefers and so offers the search fewer alternatives. The RL policy still earned its place, because the value network was trained on its games.

Compute

The networks run on GPUs and the rollouts on CPUs, asynchronously. The single-machine version used 48 CPUs and 8 GPUs. The distributed version used 1,202 CPUs and 176 GPUs.

• Q(s, a) — mean of the leaf evaluations seen below this edge, the exploitation term.
• u(s, a) — the exploration bonus; high for a large prior and few visits, fading as visits accumulate.
• P(s, a) — prior probability from the SL policy network.
• vθ — the value network; zL — the win or loss from the fast rollout.
• RL policy training moves the weights along ∂ log p_ρ(a_t | s_t) / ∂ρ, scaled by the game outcome z; the value network moves along (∂v_θ/∂θ)(z − v_θ(s)).

Worked example: if the value network says 0.30 for a leaf (on a scale of −1 to 1) and the rollout ends in a win (z = +1), the mixed value is 0.5 × 0.30 + 0.5 × 1 = 0.65.

The paper measures its pieces separately, which is why it is possible to say what each contributed.

The ablation on leaf evaluation compared a search using only the value network, only rollouts, and the mix. Each single-evaluator version was already stronger than every other Go program tested, and the mix with λ = 0.5 was clearly stronger than either. The two estimators are complementary: the value network is fast and smooth, rollouts are slow and noisy but unbiased about real endings.

Against other programs, the single-machine version won 494 of 495 games, and the distributed version won about 77 percent of games against the single-machine version. In October 2015 the distributed version beat the European champion Fan Hui 5–0 in formal games. The 4–1 win over Lee Sedol in March 2016 came after the paper appeared, with an updated system.

The system's dependencies are stated plainly, and every one of them is a target the next rung removes: it needs a database of human games to start from, it needs a separately-trained fast rollout policy to finish simulations cheaply, it runs several distinct networks, and it needs a large distributed compute setup to play at full strength. The paper is a demonstration that the combination works, not a claim that it is minimal.

What it only partly resolves is how much of the strength came from which part. The paper does ablate the leaf evaluation (value network only, rollouts only, or a mix), but the search, the learned prior and the learned evaluation all contribute, and separating them fully is the kind of ablation that a match against a human champion does not provide.

How it held up: the template held completely — a learned prior to propose, a learned evaluation to judge, and search at decision time to combine them is still the shape of every strong game-playing system. The specifics did not survive one year: rung 4 discards the human games, the rollout policy and the separate networks, and is stronger for it. The deepest consequence is elsewhere entirely. "Spend compute at decision time to improve a decision" is the idea that the reasoning ladder rediscovered for language models a long time later, and the search-improves-the-policy loop below is what reinforcement learning on reasoning traces approximates.

AlphaGo's lineage generalized fast. AlphaGo Zero dropped human data and learned Go from scratch by self-play; AlphaZero generalized the same algorithm to chess and shogi with no game-specific knowledge; MuZero learned the rules themselves, planning in a learned model of the environment. The through-line is a single algorithm — search guided by learned policy and value, improved by self-play — applied ever more generally.

Its deepest influence, though, is the idea of scaling test-time search. AlphaGo showed that spending more compute at decision time — running more MCTS simulations — makes a system stronger, separately from training. That principle, learned intuition amplified by deliberate search, is exactly what today's reasoning models revive: generate and evaluate many lines of thought before answering. From a board game to self-improving reasoners, the same bet — think longer, search guided by learned judgment — keeps paying off.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Reward, policy, search. The line from Atari to a Go engine that was never shown a human game — and the algorithm (PPO) that RLHF later borrowed wholesale.

1. 1DQN2015
2. 2PPO2017
3. 3AlphaGo2016
4. 4AlphaZero2018
5. 5MuZero2020

All 110 papers, in 19 ladders →