Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path →
Muon,
explained.
Adam has been the default optimizer for a decade, and it treats a weight matrix as a pile of unrelated numbers, tuning each one on its own. Muon takes the opposite view: a layer's weights are a matrix, and its update should be treated as one. It orthogonalizes the update — flattening its singular-value spectrum so every direction gets an equal push — using a trick that needs only matrix multiplies. The payoff is faster, more efficient training, and it helped train some of the largest recent models. Here's the geometry, and the iteration that makes it cheap.
What changes against the whole ladder: the optimiser stops being fixed
Every rung below this one varies something — the objective, the interface, the parameter count, the token count, the data quality — while holding the optimiser constant. Adam, and later AdamW, is the unexamined constant underneath rungs 1 through 6, including both sides of the Scaling-Laws-versus-Chinchilla disagreement. Rung 4's own caveat about its coefficients being specific to the optimiser settings is a hint that this was never safe.
This rung varies it. The idea is that for a weight matrix, the raw momentum update is badly conditioned — a few directions dominate it — and that orthogonalising the update before applying it makes each step move the matrix more usefully. The orthogonalisation is done with a cheap iterative approximation rather than an exact decomposition, which is what makes it affordable per step at all. One-dimensional parameters keep a conventional optimiser.
The unit that matters here is not parameters or tokens; it is loss per unit of compute. The claim is that the same target loss is reached with meaningfully less compute than a tuned AdamW baseline — which, if it holds, shifts every curve on rungs 4 and 5 rather than arguing with them.
Adam is element-wise
Adam, and most optimizers before it, are element-wise: they look at each weight independently, keep per-weight running statistics, and scale each weight's step by its own history. It works, but it ignores something real — a layer's weights aren't independent scalars, they form a matrix, and that matrix has geometric structure. Some directions in weight space the gradient has explored a lot; others barely at all.
You can see this structure through the singular values of the update. A momentum matrix with a few large singular values and many tiny ones means the update is dominated by a handful of directions — the step mostly moves along those, and the rest of the space is under-served. Element-wise scaling can't fix this, because the imbalance is a property of the matrix as a whole, not of individual entries. Muon's premise is that treating the update as a matrix, and rebalancing it, uses each step better.
Instead of scaling each weight on its own, orthogonalize the whole update matrix so every direction gets an equal-sized step — no direction dominates, none is starved.
Orthogonalize the update
Muon's core operation is to orthogonalize the momentum matrix before stepping. Concretely, if the momentum G has singular value decomposition G = U S Vᵀ, the orthogonalized update replaces the diagonal of singular values S with all ones — giving U Vᵀ, the nearest orthogonal matrix to G (its polar factor):
Keeping the directions (U, V) but flattening the magnitudes to 1 means the update pushes equally along every direction the gradient identified, instead of letting the largest singular values swamp the rest. Geometrically it's a rotation/reflection — an orthogonal transform — so it never inflates or collapses the update's scale, only re-balances it. The runnable version below shows a skewed spectrum (ratio 3:1) flattened to 1:1.
Newton-Schulz, no SVD
There's a problem: computing an SVD every optimizer step, for every weight matrix, is far too expensive — SVD is slow and awkward on accelerators. Muon sidesteps it entirely with Newton-Schulz iteration, which approximates the orthogonalization using only matrix multiplications. A short polynomial recurrence — shown here in its classic cubic form — applied a handful of times, drives the matrix's singular values toward 1 without ever forming an SVD:
Each singular value σ gets mapped by the scalar function 1.5σ − 0.5σ³, which has a stable fixed point at σ = 1 — so iterating pulls every singular value to 1 while leaving the directions untouched. (Muon itself runs a tuned quintic, X ← aX + b(XXᵀ)X + c(XXᵀ)²X with (a, b, c) = (3.4445, −4.7750, 2.0315): it converges faster but leaves singular values in a band around 1 instead of exactly at 1. The cubic is the cleanest way to see the idea, and it is what the exercise below uses.) Because it's just matmuls (with the matrix pre-scaled so the singular values start in the convergence range), it runs fast on GPUs and TPUs, cheap enough to do every step. That's the engineering that makes an idea from numerical linear algebra practical for training billion-parameter models.
Muon + Adam
Muon's orthogonalization only makes sense for things that are matrices — the hidden linear layers, which are the vast majority of a transformer's parameters. Other parameters have no matrix structure to orthogonalize:
So a Muon setup is really a hybrid: Muon for the matrices, Adam for the rest. Since the matrices dominate the parameter count, Muon governs most of the training dynamics while Adam handles the odds and ends it isn't suited to. It's a clean division of labor along the shape of each parameter.
How a Muon update works, step by step
Muon, short for MomentUm Orthogonalized by Newton–Schulz, was developed in the NanoGPT speedrun community in late 2024 and written up by Keller Jordan and collaborators in December 2024. The arXiv paper this page links, by Liu and colleagues at Moonshot AI (2025), is the follow-up that scaled it to large language models. The update below is Jordan's, with the two changes Moonshot found necessary at scale marked.
- Route each parameter. Hidden-layer weight matrices, meaning the attention projections and MLP weights, use Muon. Embeddings, the output head, biases and normalization gains use AdamW. For the vectors that is forced, since there is no matrix to orthogonalize. For the embedding and the head it is a choice, since both are matrices: Jordan reports they train better on AdamW.
- Accumulate momentum. B ← μ·B + G with μ = 0.95, where G is the minibatch gradient of the weight matrix. The default uses the Nesterov form, orthogonalizing G + μ·B rather than B itself.
- Normalize. Divide by the Frobenius norm, plus a small ε. The spectral norm never exceeds the Frobenius norm, so every singular value is now at most 1, inside the range the iteration handles.
- Orient. If the matrix has more rows than columns, transpose it, so that the Gram matrix X·Xᵀ is the smaller of the two possible.
- Run five quintic Newton–Schulz steps in bfloat16. Each step is X ← a·X + b·(X·Xᵀ)·X + c·(X·Xᵀ)^2·X with (a, b, c) = (3.4445, −4.7750, 2.0315). The coefficients are tuned for speed, not exactness: the large slope at zero pulls tiny singular values up fast, and the price is that the result is not exactly orthogonal, with singular values ending in a band of roughly 0.7 to 1.2 instead of at 1. Jordan reports that this costs nothing measurable: replacing the exact UVᵀ with a version whose singular values are spread between 0.5 and 1.5 did not hurt training.
- Scale for shape. An orthogonal m×n update has a root-mean-square entry of 1/√max(m, n), so its effective step depends on the matrix's shape. Jordan's code multiplies by √max(1, m/n). Moonshot instead multiplies by 0.2·√max(m, n), which sets the update's RMS to about 0.2, in the range of AdamW's typical updates, so AdamW's learning rate and weight decay can be reused unchanged.
- Apply, with weight decay. W ← W − η·(scaled update + λ·W). Jordan's original had no weight decay; Moonshot found that without it, weights and layer outputs kept growing over long runs, eroding Muon's advantage and pushing values beyond the range bfloat16 represents precisely.
Why orthogonalize at all
Two derivations lead to the same update. Bernstein and Newhouse show that UVᵀ is the steepest-descent direction when step size is measured in the spectral norm, a natural norm for a linear layer, since it bounds how much the layer can change its outputs. And Shampoo preconditions a gradient as (GGᵀ)^−1/4·G·(GᵀG)^−1/4; with its accumulation switched off, that is exactly UVᵀ. Muon's contribution is making the step cheap enough to run every iteration: by Jordan's estimate the Newton–Schulz matmuls add about T·m/B FLOPs relative to the forward and backward passes, for T iterations, model width m and a batch of B tokens, which is under 1% at typical language-model batch sizes.
The Muon update rule
Gt— the gradient of an m×n weight matrix;μ = 0.95; the Nesterov form feeds G + μB to the iteration.‖·‖F— the Frobenius norm; dividing by it guarantees every singular value is at most 1.(a, b, c) = (3.4445, −4.7750, 2.0315)— each step maps every singular value σ to aσ + bσ^3 + cσ^5 and leaves the singular vectors unchanged, so the iteration reshapes the spectrum and nothing else.s— Moonshot's RMS-matching scale (Jordan's version uses √max(1, m/n));λ— decoupled weight decay, Moonshot's addition.
At the extremes: σ = 0 stays 0, so a direction the gradient never touched is never created. Small values grow by about 3.4 times per step at first, so a singular value of 0.01 reaches the band in four steps. With the classic cubic coefficients (1.5, −0.5, 0), σ = 1 is an exact fixed point but small values grow only 1.5 times per step and need many more iterations: that is the trade Muon's tuned quintic makes.
What the Muon runs showed, and what is still contested
The evidence comes in three layers: Jordan's speedruns, Moonshot AI's scaling study, and Moonshot's production models.
| Setting | Reported result | Reported by |
|---|---|---|
| CIFAR-10 speedrun, to 94% accuracy | Record cut from 3.3 to 2.6 A100-seconds | Jordan and colleagues, 2024 |
| NanoGPT speedrun, to 3.28 validation loss on FineWeb | 1.35× faster than the previous record | Jordan and colleagues, 2024 |
| 1.5B-parameter transformer to GPT-2 XL level on HellaSwag | 10 hours on 8×H100, against 13.3 with AdamW | Jordan and colleagues, 2024 |
| Scaling-law sweep at compute-optimal settings | About 2× AdamW's compute efficiency | Liu and colleagues (Moonshot AI), 2025 |
| Moonlight, 16B total and 3B active, 5.7T tokens | Better quality per training FLOP than comparable AdamW-trained models | Liu and colleagues, 2025 |
| Kimi K2, about 1T total and 32B active, 15.5T tokens | Trained with MuonClip and no loss spikes | Moonshot AI, 2025 |
The shared mechanism is cheap per step, a few matrix multiplies against the layer's own math, and it held up where new optimizers usually fail: at scale. Making it hold up took fixes. Moonshot found that without weight decay Muon's advantage faded over long training, and that switching to Muon only for fine-tuning a model pre-trained with AdamW gave no clear gain. Kimi K2 needed MuonClip, which rescales the query and key weights whenever the largest attention logit passes a threshold, because attention logits grew too large during training without it.
The contested part
Every comparison above is against an AdamW baseline tuned by the people proposing the alternative, and an optimizer comparison is only as fair as that baseline. A 2025 Stanford benchmark by Wen, Hall, Ma and Liang tuned ten optimizers carefully at each scale and found that matrix-based methods such as Muon were still the fastest, but that their speedup over well-tuned AdamW shrank as models grew, to roughly 1.1× near a billion parameters. Read the speedups as properties of particular runs, not as constants.
What the authors admit, and what happened next
The honest summary of this rung's status is that it is the least settled page on this ladder, and it is placed last for that reason rather than only by date. The reported efficiency gains come from a limited set of runs; the method has a shape that makes clean comparison hard, because an optimiser comparison is only as good as the tuning of the baseline, and a well-tuned AdamW baseline is a moving target. Reported speed-ups should be read as "on these runs, against this baseline", not as a constant.
What has happened since is the strongest evidence available: the method has been used in real frontier-scale training rather than only in papers — most visibly in the Kimi K2 line — which is a much higher bar than a benchmark table, because a lab betting a full pre-training run on an optimiser is a lab that has convinced itself internally.
The open question worth holding onto after reading rungs 4 and 5: if changing the optimiser changes the exponents, then the compute-optimal ratio is not a property of the architecture and the data, it is a property of the whole training setup — and the ~20:1 rule is a measurement of one configuration rather than a law. Nobody has re-derived Chinchilla's curves under this optimiser in public. That is the experiment this ladder is missing.
Flatten the singular-value spectrum
Muon's heart is orthogonalization, and the textbook cubic Newton-Schulz map 1.5σ − 0.5σ³ (Muon's own tuned quintic is a faster cousin) shows how. This confirms an orthogonal (rotation) matrix satisfies RᵀR = I, that one Newton-Schulz step moves a small singular value (0.5 → 0.6875) toward 1, that iterating converges any singular value in range to 1, and — the point — that a skewed spectrum with a 3:1 ratio between its largest and smallest singular value comes out 1:1 after orthogonalization. That equalization is what makes the update push equally in all directions. Change the singular values and watch them converge.
Write the body of orthogonalize_sv. Everything else in the file is
already written; this is the function the paper is about. Press ▶ Run to check it against
the 5 assertions below — they start red.
- Matmul + transpose satisfy the identity check
- An orthogonal matrix satisfies R^T R = I
- One Newton-Schulz step moves a small singular value toward 1
- Iterating Newton-Schulz converges singular values to 1
- Muon equalizes a skewed singular-value spectrum
Its place
Muon is part of a renewed interest in optimizers that respect a network's structure rather than treating parameters as an undifferentiated vector. The line of thought — Shampoo and other matrix/second-order-flavored methods, and now Muon — asks whether the geometry of weight matrices should shape the update, and answers yes, if you can compute it cheaply enough. Newton-Schulz is what made "cheaply enough" true for orthogonalization.
Its broader lesson echoes Adam's: a well-chosen transformation of the raw gradient can matter as much as the gradient itself. Adam's was per-parameter normalization; Muon's is per-matrix orthogonalization. Both take the noisy, imbalanced signal that backprop produces and reshape it into something the model can step along more effectively — a reminder that how you use the gradient is a frontier as live as what the gradient is.
MuonClip, the variant used at scale, adds a clipping step (on the attention logits, via QK-clip) to prevent the rare instabilities that can arise when orthogonalized updates meet very large activations — the stability patch that let Muon run a trillion-parameter model spike-free.
Questions people ask about Muon
What Newton-Schulz coefficients does Muon use, and why those values?
Muon's reference implementation runs five iterations of a quintic polynomial, X becomes aX + b(XXᵀ)X + c(XXᵀ)²X, with a = 3.4445, b = −4.7750 and c = 2.0315, in bfloat16, after dividing the matrix by its Frobenius norm. Per singular value the map is aσ + bσ³ + cσ⁵. The large first coefficient makes tiny singular values grow quickly, so five steps are enough; the cost is that the polynomial does not converge to exactly 1 but leaves singular values in a band of roughly 0.7 to 1.2. Keller Jordan reported that this inexactness does not hurt training, which is why speed was preferred over precision. The classic cubic iteration, 1.5X − 0.5XXᵀX, converges exactly but needs many more steps.
Is Muon better than AdamW?
On the runs reported so far, usually yes, by an amount that depends on scale and on how well AdamW is tuned. Jordan's speedruns showed clear wall-clock gains, such as reaching a 1.5-billion-parameter model's target in 10 rather than 13.3 hours on eight H100s. Moonshot AI's scaling-law experiments put Muon at about twice AdamW's compute efficiency, and Moonshot trained Moonlight and the trillion-parameter Kimi K2 with Muon-based optimizers. A 2025 Stanford benchmark by Wen, Hall, Ma and Liang, which tuned every optimizer carefully, found matrix optimizers like Muon fastest at small scale but with an advantage over AdamW that shrank to roughly 1.1 times near a billion parameters. A real but scale-dependent gain is the fair summary.
Why does Muon need weight decay at large scale?
Moonshot AI found that the original Muon, which had no weight decay, worked well early in training but lost its edge over AdamW in long runs. Without decay, the weights and layer outputs kept growing in magnitude, which hurt quality and pushed values toward the edge of what bfloat16 represents precisely. Adding decoupled weight decay, the same kind AdamW uses, fixed it: the weights shrink by a factor of one minus the learning rate times the decay rate each step, in addition to the orthogonalized update. Together with rescaling the update so its root-mean-square matches AdamW's, this let Moonshot reuse AdamW's learning rate and weight-decay settings directly when switching optimizers.
Why does Muon use AdamW for the embedding and output layers?
It is a choice, not a limitation of shape. Biases and normalization gains are vectors, so there is nothing to orthogonalize and they need an element-wise optimizer. But the token embedding and the output head are matrices and could be orthogonalized. Jordan's write-up keeps them on AdamW because they trained better that way in his experiments. One plausible reason is that these layers do not behave like generic hidden linear maps: an embedding update touches only the rows of tokens present in the batch, and the output layer shapes the logits directly. Moonshot AI kept the same split when it scaled Muon to large models.
How much overhead does Muon's Newton-Schulz iteration add?
Little, for large-batch training. Each of the five iterations is a few matrix multiplies on the weight-shaped update, and the cost is paid once per optimizer step rather than once per token. Jordan estimated the extra floating-point work, relative to the forward and backward passes, at about T times m divided by B, where T is the number of iterations, m the model width and B the batch size in tokens. That is about 0.7% for the NanoGPT speedrun and about 0.5% for a run at Llama-405B scale. The overhead grows with small batches or very wide matrices, and in distributed training gathering each full matrix adds communication, which Moonshot's Distributed Muon addresses.
4 questions to test what you just read
Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.
Apply one quintic step with (3.4445, −4.7750, 2.0315) to σ = 0.1 and to σ = 1.0. Which moves toward 1, and which is pushed down?
Show answer
σ = 0.1: 0.34445 − 0.00478 + 0.00002 ≈ 0.34, more than tripled. σ = 1.0: 3.4445 − 4.7750 + 2.0315 = 0.70, pulled down. The polynomial is not designed to fix 1 exactly; it maps everything into a band around it.
An orthogonal 4096×4096 update has every singular value equal to 1. What is its RMS entry size, and what does Moonshot's 0.2·√max(m, n) scaling make it?
Show answer
‖O‖F^2 = 4,096, one per singular value, so RMS = √(4,096 / 4,096^2) = 1/64 ≈ 0.0156. Multiplying by 0.2 × √4,096 = 12.8 gives RMS 0.2, matching AdamW's typical update size so its learning rate carries over.
Roughly how much Newton–Schulz overhead does Muon add for model width m = 4096, a 4M-token batch and T = 5 iterations?
Show answer
T·m/B = 5 × 4,096 / (4 × 10^6) ≈ 0.5% of the forward and backward FLOPs. Halve the batch and the overhead doubles, which is why Muon is cheapest in exactly the large-batch regime of pre-training.
Why is dividing by the Frobenius norm enough to keep the five steps from diverging?
Show answer
The spectral norm, the largest singular value, never exceeds the Frobenius norm, the root of the sum of all squared singular values. After the division every σ ≤ 1, and the quintic maps that whole range into roughly [0, 1.2] and keeps it there, so no singular value can blow up.
Pre-training, and the scaling laws
The ladder where rung 5 overturns rung 4. Read `scaling-laws` before `chinchilla` — the second measures the same axis in the same units and gets a different answer.
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Paper Breakdowns
- Attention Is All You NeedThe paper that introduced the Transformer — the architecture underneath GPT, BERT, Claude, and Gemini. Self-attention, Query/Key/Value, multi-head attention, positional encoding, and the real BLEU results, explained in depth with analogies and animated visuals, not just equations.Read →
- Word2VecHow words became math. Dense word vectors learned from the company a word keeps, so king − man + woman lands near queen. Skip-gram, CBOW, the distributional hypothesis, and the origin of every embedding model powering search and RAG.Read →
- ResNetThe skip connection that made deep learning deep. Why stacking layers made networks worse, and how residual learning let them reach 150+ layers and win ImageNet — the identity shortcut every Transformer still uses inside every block.Read →
- Vision Transformer (ViT)When an image became 16×16 words. Cut a picture into patches, treat each patch as a token, and feed them to a plain Transformer — no convolutions. Patch embeddings, the class token, and why data scale lets attention beat the CNNs that ruled computer vision for a decade.Read →
- Latent DiffusionThe paper behind Stable Diffusion. Compress an image into a small latent space with a VAE, run the diffusion denoising there instead of on pixels for a tens-of-times efficiency win, and steer it with text via cross-attention. The architecture that put text-to-image generation on consumer GPUs.Read →
- DDPMThe original denoising diffusion math — the paper that made diffusion work. A fixed forward process adds Gaussian noise over many steps; a U-Net learns the reverse. The masterstroke: train the network to simply predict the added noise with a plain MSE loss. The foundation under every modern image generator.Read →