MuZero (Schrittwieser et al., DeepMind, 2020) is a reinforcement-learning algorithm that plans without being given the rules of the environment. It learns its own model with three networks — a representation network that encodes observations into a latent state, a dynamics network that predicts the next latent state and reward from a state and action, and a prediction network that outputs a policy and value — and runs Monte Carlo Tree Search entirely inside that learned model. It matched AlphaZero on Go, chess, and shogi and also mastered Atari, all without a simulator.

How does MuZero differ from AlphaZero?

AlphaZero is given a perfect simulator — it knows the rules, so its tree search can apply real moves and see real next states. MuZero is not given the rules; it learns a model of the environment's dynamics and plans in that learned model. This lets the same algorithm work in domains like Atari where the exact rules are unknown or complex, which AlphaZero could not handle.

What does it mean that MuZero's model is 'value-equivalent'?

MuZero's learned model is not trained to reconstruct observations or predict exact next frames. It is trained only so that the quantities that matter for planning — the predicted reward, value, and policy along imagined rollouts — match reality. The latent states can be arbitrary internal representations, as long as planning through them yields correct decisions. This 'value-equivalent' or planning-useful model is far easier to learn than an accurate simulator.

How does MuZero plan without a simulator?

It runs Monte Carlo Tree Search using its dynamics network in place of a simulator. Starting from the latent state of the current observation, each simulation unrolls the dynamics network — repeatedly applying (state, action) -> (next state, reward) — to imagine future latent states, evaluates leaves with the prediction network's value, and backs up the results. Because the whole rollout happens in the learned latent space, no real environment or rulebook is required.

Paper Breakdowns / MuZero

Paper 72~11 min readNature 2020worked math + runnable code

Paper Breakdown

MuZero,
explained.

AlphaZero could master Go, chess, and shogi — but only because it was handed a perfect rulebook to plan against. Take away the rules and it's helpless, which is why it couldn't play a video game where nobody wrote down the physics. MuZero removed that crutch: it learns its own model of how the world works and plans inside that imagined model. The twist is what it learns — not an accurate simulator, but just enough of one to make good decisions. One algorithm then conquered board games and Atari. Here's the model it builds, and why it doesn't need to be right about everything.

Video breakdown

The animated walkthrough is in production.

Read the full breakdown below in the meantime ↓

Prefer video? Browse published breakdowns →

Planning without rules Three networks Search in latent space Useful, not accurate Run it yourself One algorithm, many games Why it matters

Planning without rules

AlphaZero plans by simulating: its tree search applies a real move and asks the rules engine "what's the board now?" That requires a perfect model of the environment — you must know the rules exactly. For board games that's fine; the rules are given. But most interesting problems don't hand you a simulator. An Atari game's dynamics are buried in emulator code; the real world has no rulebook at all. Planning methods that need a perfect model are stuck at the edge of games with known rules.

MuZero's move is to learn the model. Instead of being told how the world transitions, it trains a network to predict transitions from experience, and then plans against that. The catch it solves elegantly is that learning an accurate simulator — one that predicts exact next frames — is hard and, it turns out, unnecessary. You don't need to predict the world perfectly; you only need to predict the things that matter for choosing actions. That relaxation is what makes the whole thing work.

The one-sentence version

Learn a model that predicts reward, value, and policy along imagined rollouts — not exact observations — and run tree search inside that learned model, so planning needs no rules.

Three networks

MuZero's model is three learned functions that work together:

h (representation): observation → latent state s₀
g (dynamics): (state, action) → next state, reward
f (prediction): state → policy, value

The representation network h encodes the current observation into an abstract latent state — the starting point for planning. The dynamics network g is the learned simulator: give it a latent state and an action and it returns the next latent state and the reward, so you can unroll the future without ever touching the real environment. The prediction network f reads a latent state and outputs a policy (which moves look good) and a value (how good is this state) — the same signals AlphaGo's policy and value nets provided, but now over learned states. The runnable version below unrolls g to imagine a trajectory.

Search in latent space

Planning is MCTS, exactly as in AlphaZero — but the simulator is the dynamics network. From the latent state of the current observation, each simulation walks down the tree, and to expand a new node it calls g(state, action) to imagine the next latent state and its reward, evaluates the leaf with f's value, and backs the result up. Every step of this happens inside the learned latent space; no real moves are played, no rulebook consulted.

The value that gets backed up is an n-step bootstrapped return: the discounted sum of the rewards imagined along the rollout, plus the discounted value of the final state. This is standard reinforcement-learning bookkeeping, now computed over a trajectory the model dreamed up:

G = r₁ + γ r₂ + γ² r₃ + … + γ^k v(s_k)

Rewards close to now count fully; distant ones are discounted by γ^t; and the tail beyond the rollout is estimated by the value network. The runnable below computes exactly this — including the case where, with no rewards, the return is just the discounted leaf value.

Useful, not accurate

Here is MuZero's deepest idea. Its model is never trained to reconstruct observations. The latent states can be any internal representation the network likes — they need not correspond to anything human-readable, and MuZero never tries to render a next frame. The model is trained only so that three planning-relevant quantities match reality:

What the model is trained to get right

RewardThe dynamics net's predicted reward matches the real reward observed.

ValueThe prediction net's value matches the actual return (from search + real outcomes).

PolicyThe prediction net's policy matches the improved policy the search produced.

Not: pixelsNo loss for reconstructing observations — the latent is free to be abstract.

This is value-equivalence: a model is good enough if planning through it gives the right decisions, regardless of whether it "understands" the environment in any other sense. Learning a full simulator is hard; learning a model that's merely useful for planning is much easier — and sufficient. It's a striking lesson that a model can be profoundly wrong about the world in every way that doesn't affect the choices it needs to make, and still plan perfectly.

RUN IT YOURSELF

Unroll the model, bootstrap the value

MuZero's planning math in a few lines. The representation net encodes an observation into a fixed latent dimension; the dynamics net unrolls the future in that latent space — three actions become four latent states and three predicted rewards, no simulator touched; and the value backed up is the n-step bootstrapped return: discounted rewards along the rollout (2.17 for these) plus the discounted leaf value, which with no rewards reduces to pure bootstrap (γ³·v = 1.46). Change the actions, rewards, or discount and watch the imagined trajectory and its value move.

CPython · WebAssembly

# MuZero plans in a LEARNED model. h: obs -> latent state; g: (state, action) ->
# next state, reward (unrolls the future in latent space); f: state -> policy, value.
# MCTS runs on g's imagined rollouts; the value backed up is an n-step return:
#   G = r1 + gamma*r2 + ... + gamma^k * v(leaf)

def representation(obs, latent_dim):
    return [sum(obs) / len(obs)] * latent_dim      # toy: pool the observation to a latent

def dynamics(state, action):                        # toy learned transition + reward
    next_state = [x + action * 0.1 for x in state]
    reward = action                                 # a stand-in learned reward
    return next_state, reward

def rollout(s0, actions):                           # unroll the learned model in latent space
    states, rewards, s = [s0], [], s0
    for a in actions:
        s, r = dynamics(s, a)
        states.append(s)
        rewards.append(r)
    return states, rewards

def bootstrapped_return(rewards, leaf_value, gamma):
    G = 0.0
    for t, r in enumerate(rewards):
        G += (gamma ** t) * r                       # discounted rewards along the rollout
    G += (gamma ** len(rewards)) * leaf_value       # + bootstrap on the leaf value
    return G

s0 = representation([1.0, 3.0, 2.0], 4)
states, rewards = rollout(s0, [1, 1, 1])
print(f"latent s0 (dim {len(s0)}): {s0}")
print(f"rollout: {len(states)} states, rewards {rewards}\n")
print(f"n-step bootstrapped return: {bootstrapped_return([1, 0, 1], 0.5, 0.9):.4f}")
print(f"no rewards -> pure bootstrap: {bootstrapped_return([0, 0, 0], 2.0, 0.9):.4f}")

One algorithm, many games

MuZero's result was breadth without loss of depth:

Domain	Result
Go, chess, shogi	Matched AlphaZero's superhuman play — without being given the rules.
Atari	State-of-the-art on the Atari benchmark, a domain AlphaZero couldn't touch.
One algorithm	The same method mastered both discrete board games and pixel-based video games.
Model-based sample efficiency	Planning in a learned model brought strong data efficiency.

The significance is generality. AlphaZero was a triumph confined to games with known rules; MuZero showed the same plan-with-learned-value approach works whenever you can only experience an environment, not consult its rulebook — which is almost everything. It unified two families that had been separate: powerful planning (from the AlphaZero line) and model-free learning from raw experience (from the Atari line).

Why it matters

MuZero pushed model-based reinforcement learning into the mainstream by making the model learnable and useful rather than hand-specified and accurate. Its descendants extended it to continuous control, offline data, and beyond, and its value-equivalence principle — learn only what planning needs — reshaped how people think about world models. Real-world deployments followed, from data-center cooling to video compression, wherever planning against a learned model beats reacting.

Its enduring idea reframes what a "model of the world" is for. We tend to assume a good world model should predict the world faithfully — render the next frame, get the physics right. MuZero argues that's the wrong target: a model is good if decisions made through it are good, and that is a far lower, far more learnable bar. That shift — from predicting everything to predicting only what changes your action — is one of the more quietly profound ideas in modern RL, and it echoes in every system that plans against a learned approximation of reality.

Worth knowing

Because MuZero's latent states are unconstrained, they can encode useful abstractions the real state doesn't make explicit — the model is free to represent whatever helps it plan, which is part of why the learned model can be both compact and effective.

Frequently asked

Quick answers

What is MuZero in one line?

A planning agent that learns its own model (representation, dynamics, prediction networks) and runs MCTS inside that learned latent model — no rules needed.

How is it different from AlphaZero?

AlphaZero is given a perfect simulator (the rules); MuZero learns a model from experience and plans in it, so it works where rules are unknown (like Atari).

What is value-equivalence?

The model is trained only so predicted reward, value, and policy match reality — not to reconstruct observations. A model that's useful for planning suffices.

Why does it matter?

It unified planning and learning from raw experience, mastering board games and Atari with one algorithm, and reshaped how world models are conceived.

Finished this one? 0 / 101 Paper Breakdowns done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

ML Foundations

MuZero,
explained.

Planning without rules

Three networks

Search in latent space

Useful, not accurate

Unroll the model, bootstrap the value

One algorithm, many games

Why it matters

Quick answers

What is MuZero in one line?

How is it different from AlphaZero?

What is value-equivalence?

Why does it matter?

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

MuZero,explained.

Planning without rules

Three networks

Search in latent space

Useful, not accurate

Unroll the model, bootstrap the value

One algorithm, many games

Why it matters

Quick answers

What is MuZero in one line?

How is it different from AlphaZero?

What is value-equivalence?

Why does it matter?

Go deeper

AlphaGo →

Reinforcement Learning →

STaR →

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Get the next one in your inbox.

MuZero,
explained.