Muon,
explained.

Adam, and most optimizers before it, are element-wise: they look at each weight independently, keep per-weight running statistics, and scale each weight's step by its own history. It works, but it ignores something real — a layer's weights aren't independent scalars, they form a matrix, and that matrix has geometric structure. Some directions in weight space the gradient has explored a lot; others barely at all.

You can see this structure through the singular values of the update. A momentum matrix with a few large singular values and many tiny ones means the update is dominated by a handful of directions — the step mostly moves along those, and the rest of the space is under-served. Element-wise scaling can't fix this, because the imbalance is a property of the matrix as a whole, not of individual entries. Muon's premise is that treating the update as a matrix, and rebalancing it, uses each step better.

Muon's core operation is to orthogonalize the momentum matrix before stepping. Concretely, if the momentum G has singular value decomposition G = U S Vᵀ, the orthogonalized update replaces the diagonal of singular values S with all ones — giving U Vᵀ, the nearest orthogonal matrix to G (its polar factor):

Keeping the directions (U, V) but flattening the magnitudes to 1 means the update pushes equally along every direction the gradient identified, instead of letting the largest singular values swamp the rest. Geometrically it's a rotation/reflection — an orthogonal transform — so it never inflates or collapses the update's scale, only re-balances it. The runnable version below shows a skewed spectrum (ratio 3:1) flattened to 1:1.

There's a problem: computing an SVD every optimizer step, for every weight matrix, is far too expensive — SVD is slow and awkward on accelerators. Muon sidesteps it entirely with Newton-Schulz iteration, which approximates the orthogonalization using only matrix multiplications. A short polynomial recurrence, applied a handful of times, drives the matrix's singular values toward 1 without ever forming an SVD:

Each singular value σ gets mapped by the scalar function 1.5σ − 0.5σ³, which has a stable fixed point at σ = 1 — so iterating pulls every singular value to 1 while leaving the directions untouched. Because it's just matmuls (with the matrix pre-scaled so the singular values start in the convergence range), it runs fast on GPUs and TPUs, cheap enough to do every step. That's the engineering that makes an idea from numerical linear algebra practical for training billion-parameter models.

Muon's orthogonalization only makes sense for things that are matrices — the hidden linear layers, which are the vast majority of a transformer's parameters. Other parameters have no matrix structure to orthogonalize:

So a Muon setup is really a hybrid: Muon for the matrices, Adam for the rest. Since the matrices dominate the parameter count, Muon governs most of the training dynamics while Adam handles the odds and ends it isn't suited to. It's a clean division of labor along the shape of each parameter.

Muon's practical case is efficiency: it reaches a given loss in fewer steps or less compute than Adam on the matrix parameters, and it scales:

The scaling result is notable because new optimizers often work at small scale and fall apart at large. Muon's adoption in a trillion-parameter training run — paired with clipping tricks to keep it stable — is real evidence that treating updates as matrices, not scalar bags, holds up where it matters.

Muon is part of a renewed interest in optimizers that respect a network's structure rather than treating parameters as an undifferentiated vector. The line of thought — Shampoo and other matrix/second-order-flavored methods, and now Muon — asks whether the geometry of weight matrices should shape the update, and answers yes, if you can compute it cheaply enough. Newton-Schulz is what made "cheaply enough" true for orthogonalization.

Its broader lesson echoes Adam's: a well-chosen transformation of the raw gradient can matter as much as the gradient itself. Adam's was per-parameter normalization; Muon's is per-matrix orthogonalization. Both take the noisy, imbalanced signal that backprop produces and reshape it into something the model can step along more effectively — a reminder that how you use the gradient is a frontier as live as what the gradient is.