MuZero,
explained.

AlphaZero's search needs to know what each move does. MuZero's change is to learn that, which turns a board-game algorithm into a general one. Three networks replace the given simulator: a representation network mapping an observation to a latent state, a dynamics network mapping a latent state and an action to the next latent state and the immediate reward, and a prediction network reading a policy and a value off a latent state. Tree search then runs entirely inside the latent space, never touching the real environment or its rules.

The decisive design choice is what the learned model is not asked to do. It is never trained to reconstruct observations, predict pixels, or be accurate about the world in any sense a human would recognise. It is trained only so that the policy, value and reward predictions made after unrolling it match what actually happened. A model that is useful for planning and unfaithful as a simulator is explicitly acceptable — the latent state is free to discard everything that does not affect the decision, which is most of the screen.

Stated in rung 4's units: it matches AlphaZero on Go, chess and shogi — where AlphaZero has the true rules and MuZero does not, so parity is the claim — and then reaches state of the art on Atari, where AlphaZero cannot be run at all because no rule model exists. Same algorithm, one dependency fewer, one whole domain more.

AlphaZero plans by simulating: its tree search applies a real move and asks the rules engine "what's the board now?" That requires a perfect model of the environment — you must know the rules exactly. For board games that's fine; the rules are given. But most interesting problems don't hand you a simulator. An Atari game's dynamics are buried in emulator code; the real world has no rulebook at all. Planning methods that need a perfect model are stuck at the edge of games with known rules.

MuZero's move is to learn the model. Instead of being told how the world transitions, it trains a network to predict transitions from experience, and then plans against that. The catch it solves elegantly is that learning an accurate simulator — one that predicts exact next frames — is hard and, it turns out, unnecessary. You don't need to predict the world perfectly; you only need to predict the things that matter for choosing actions. That relaxation is what makes the whole thing work.

MuZero's model is three learned functions that work together:

The representation network h encodes the current observation into an abstract latent state — the starting point for planning. The dynamics network g is the learned simulator: give it a latent state and an action and it returns the next latent state and the reward, so you can unroll the future without ever touching the real environment. The prediction network f reads a latent state and outputs a policy (which moves look good) and a value (how good is this state) — the same signals AlphaGo's policy and value nets provided, but now over learned states. The runnable version below unrolls g to imagine a trajectory.

Planning is MCTS, exactly as in AlphaZero — but the simulator is the dynamics network. From the latent state of the current observation, each simulation walks down the tree, and to expand a new node it calls g(state, action) to imagine the next latent state and its reward, evaluates the leaf with f's value, and backs the result up. Every step of this happens inside the learned latent space; no real moves are played, no rulebook consulted.

The value that gets backed up is an n-step bootstrapped return: the discounted sum of the rewards imagined along the rollout, plus the discounted value of the final state. This is standard reinforcement-learning bookkeeping, now computed over a trajectory the model dreamed up:

Rewards close to now count fully; distant ones are discounted by γ^^t; and the tail beyond the rollout is estimated by the value network. The runnable below computes exactly this — including the case where, with no rewards, the return is just the discounted leaf value.

Here is MuZero's deepest idea. Its model is never trained to reconstruct observations. The latent states can be any internal representation the network likes — they need not correspond to anything human-readable, and MuZero never tries to render a next frame. The model is trained only so that three planning-relevant quantities match reality:

This is value-equivalence: a model is good enough if planning through it gives the right decisions, regardless of whether it "understands" the environment in any other sense. Learning a full simulator is hard; learning a model that's merely useful for planning is much easier — and sufficient. It's a striking lesson that a model can be profoundly wrong about the world in every way that doesn't affect the choices it needs to make, and still plan perfectly.

MuZero keeps AlphaZero's loop of search, self-generated data and training. What it changes is where the search gets its states from, and therefore what the training data has to teach.

1. Encode the history. The representation network h reads recent observations (for Atari, the last 32 frames at 96×96 along with the actions taken; for board games, the last 8 positions) and produces the starting latent state s^⁰. After h and after every dynamics step, the latent state is rescaled to the range 0 to 1, so successive steps stay on a comparable scale.
2. Search in latent space. The root is s^⁰. Each simulation chooses actions down the tree with the same prior-plus-value rule as before. At a new node, the dynamics network g takes the parent latent state and the chosen action and returns the next latent state and a predicted reward. The prediction network f reads that new state and returns policy logits and a value. Nothing ever touches the real environment during search.
3. Normalize values inside the tree. Atari rewards have no fixed range, so the search keeps the minimum and maximum value it has seen in the tree and rescales Q into 0 to 1 before adding the exploration bonus.
4. Act and record. The agent plays from the root visit counts: 800 simulations per move for board games and 50 for Atari. It stores the real observations, actions, rewards, the visit-count distribution and the search's root value.
5. Sample a position and unroll K = 5 steps. Training picks a stored position t, runs h on the observation history to get s^⁰, then feeds the real next five actions through g, calling f at every step. That produces five predicted rewards, six policies and six values.
6. Build targets from what really happened. The policy target at step k is the stored search distribution for position t + k. The reward target is the real reward. The value target is an n-step return: the next n real rewards, discounted, plus the discounted search value n steps ahead. Atari uses n = 10. For board games the value target is simply the final result.
7. Backpropagate through the whole unroll. The summed loss sends gradients into f, g and h together. The gradient entering each dynamics step is halved, so a long unroll does not multiply it. No term asks the latent state to reproduce a screen.
8. Handle large values with a categorical output (Atari). Rewards and values are not regressed directly. They are first compressed with a signed square-root-like transform, then predicted as a distribution over 601 bins with a cross-entropy loss. That keeps one scale of loss usable across games where returns differ by orders of magnitude.

Reanalyze

Stored search results become stale as the network improves. In the Reanalyze variant, old trajectories are fetched again, the current network re-runs the search on them, and the refreshed policy and value become the training targets. It spends extra compute to squeeze more learning from each real step, which matters when real environment steps are the expensive part.

• r^k, v^k, p^k — reward, value and policy predicted after k imagined steps; at k = 0 there is no reward.
• u — the real reward observed from the environment.
• π — the visit-count policy the search produced at that position.
• νt+n — the search's value estimate n steps ahead, used to finish the return; n = 10 on Atari.
• K — the unroll length, 5 in the paper.

Worked example: with γ = 0.997, n = 2, real rewards 1 and 0 and a search value of 5 two steps later, z = 1 + 0.997 × 0 + 0.997^² × 5 ≈ 5.97. The same target is used for the value head at that step of the unroll.

The question was whether a model that is never given the rules can match one that is. In chess and shogi, MuZero matched AlphaZero's strength. In Go it slightly exceeded AlphaZero, while using less computation per search node, which suggests the hidden state is storing useful information rather than recomputing it. On Atari, where AlphaZero cannot run at all because no rule model exists, MuZero set a new state of the art in both mean and median human-normalized score across 57 games, in the budget of 200 million frames per game.

The paper includes an experiment on how much planning helps. On Ms. Pac-Man, giving the search more simulations per move raised performance, and a network trained with a small number of simulations still improved when searched more deeply at test time. A second experiment in Go compared strength as thinking time grew and found it kept rising. Both support the premise that the learned model is accurate enough to plan with, even though it was never trained to predict pixels. The Reanalyze variant was built for the other axis, using fewer real environment frames per unit of strength.

The costs are acknowledged and they are real: many simulations per move makes acting expensive, training unrolls the dynamics network for several steps on every example, and a learned model can be exploited by its own search — if the dynamics network is wrong in some corner in an optimistic direction, search will find that corner and plan into it, a failure mode a perfect simulator cannot have. Value targets are bootstrapped over n steps rather than taken from game outcomes, which is what makes non-terminating environments tractable and also what makes the target noisier.

The part left genuinely open is interpretability of the latent: because the model is trained only for planning utility, there is no guarantee — and no easy test — that its internal state corresponds to anything in the environment. That is the intended design and it makes diagnosis hard.

How it held up: the principle survived and spread. Learn a model in whatever representation makes planning work, not one that reconstructs the world is now the default position in model-based reinforcement learning, and the successors that followed pushed it into sample-efficiency regimes and real-world control. Read against the top of this site's language ladders, this rung is the strongest available statement of a claim the reasoning models make implicitly: a system can plan usefully in a representation that nobody can read.

MuZero pushed model-based reinforcement learning into the mainstream by making the model learnable and useful rather than hand-specified and accurate. Its descendants extended it to continuous control, offline data, and beyond, and its value-equivalence principle — learn only what planning needs — reshaped how people think about world models. Real-world deployments followed, notably video compression, wherever planning against a learned model beats reacting.

Its enduring idea reframes what a "model of the world" is for. We tend to assume a good world model should predict the world faithfully — render the next frame, get the physics right. MuZero argues that's the wrong target: a model is good if decisions made through it are good, and that is a far lower, far more learnable bar. That shift — from predicting everything to predicting only what changes your action — is one of the more quietly profound ideas in modern RL, and it echoes in every system that plans against a learned approximation of reality.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Reward, policy, search. The line from Atari to a Go engine that was never shown a human game — and the algorithm (PPO) that RLHF later borrowed wholesale.

1. 1DQN2015
2. 2PPO2017
3. 3AlphaGo2016
4. 4AlphaZero2018
5. 5MuZero2020

All 110 papers, in 19 ladders →