Learn from
nothing.

Game-playing AI had always leaned on human knowledge: opening books, handcrafted evaluation functions, decades of grandmaster games. AlphaGo beat the world champion at Go but still bootstrapped from human games and Go-specific features. AlphaZero (Silver et al., 2017) stripped all of that away. It's given only the rules — no human data, no opening theory, no domain heuristics — and learns entirely by self-play: the current network plays games against itself, and the outcomes make it better. The same algorithm, unchanged, mastered Go, chess, and shogi.

The engine is a marriage of two things neural networks and classical AI each do well. A single deep network looks at a position and outputs fast intuition: a policy (which moves look good) and a value (who's likely to win). A Monte Carlo Tree Search then uses that intuition to look ahead deliberately, exploring lines of play. Search produces better moves than the raw network; those better moves train the network; the improved network makes search better still. Round and round, from nothing to superhuman.

The network f(s) = (p, v) maps a board state s to a policy vector p (a probability for each legal move) and a scalar value v ∈ [−1, 1] (expected outcome from this position). That's the intuition: instant, but shallow — it hasn't looked ahead. MCTS supplies the depth by building a search tree, one simulation at a time:

A key simplification over older Monte Carlo methods: AlphaZero doesn't play out random games to the end to estimate a position. The network's value head is the evaluation, and the policy head focuses the search on plausible moves, so far fewer simulations are needed. After running many simulations from the root, the visit counts over the root's moves form an improved policy π — sharper than the network's raw p, because it reflects look-ahead. That improved policy is both what AlphaZero plays and what it trains toward.

The heart of the search is how it decides which move to explore next at each node. AlphaZero uses PUCT (Predictor + Upper Confidence bounds for Trees): pick the action maximizing an exploitation term plus an exploration bonus.

Q = mean value seen for move a (exploit what's working). P = the network's policy prior (steer toward good-looking moves). N = visit counts; the √(parent visits)/(1+child visits) shape makes unvisited moves attractive early, then fades. c_puct sets the exploration strength.

Read the two terms. Q(s,a) is exploitation — the average result this move has produced so far, so proven-good moves rise. U(s,a) is exploration — large when a move has a high policy prior P but few visits N, and it shrinks as visits accumulate. Early in the search, unvisited moves with strong priors get tried; as evidence piles up, the 1 + N denominator damps the bonus and attention concentrates on whatever actually has the best Q. The network's prior P steering the bonus is what makes the whole search tractable in games with enormous branching factors — you don't waste simulations on moves the network considers absurd.

Each backup updates a node's value as a running mean of the outcomes seen through it:

A new evaluation v is folded into the existing average Q over N prior visits. Over many simulations, Q converges to the search's estimate of the move's value.

And the network is trained to match what search and self-play reveal — the improved policy π (from visit counts) and the game's actual outcome z ∈ {−1, 0, +1}:

The value head is regressed toward the real outcome z; the policy head is pushed (cross-entropy) toward the search-improved policy π; a weight-decay term regularizes. One network, two heads, one combined loss.

That's the whole system: self-play generates games; MCTS with PUCT turns the network's rough intuition into strong moves and a sharpened policy; the network trains on those improved policies and final outcomes; the better network makes the next round of search stronger. The runnable version below implements the PUCT score, the move selection, and the value backup — the decision-making core of the search.

AlphaZero was a landmark for general, knowledge-free reinforcement learning:

The caveats are worth stating. AlphaZero needs a perfect simulator — it must know the rules to run search — so it doesn't directly apply to messy real-world environments where dynamics are unknown (the gap MuZero later closed). Training also consumed very large amounts of compute. But as a demonstration that a single, general recipe could reach superhuman skill in hard planning problems with no human knowledge, it was decisive.

AlphaZero distilled game-playing AI into a clean, reusable idea: pair a network that supplies learned intuition with a search that supplies deliberate lookahead, and let self-play improve both. Its direct descendant MuZero removed the last piece of hand-given knowledge — the rules — by learning a model of the environment's dynamics and planning inside it, extending the recipe to Atari and settings where the simulator isn't known.

The influence reaches well past board games. The "combine fast learned policies with slow deliberate search" pattern echoes across reinforcement learning — it complements value-based methods like DQN and policy-gradient methods like PPO by adding explicit planning on top of learned evaluation. And the deeper theme — that test-time search over a learned model can unlock capabilities the base model lacks on its own — resurfaced in modern language models, where letting a model deliberate, branch, and evaluate its own reasoning at inference (rather than answering in one shot) mirrors exactly what MCTS does over a game tree. AlphaZero's marriage of intuition and search remains one of AI's most quietly generative ideas.