Muon is Scalable for LLM Training
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2502.16982
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{optimizersmuonscalable2025,
title = {{Muon is Scalable for LLM Training}},
author = {Jingyuan Liu and Jianlin Su and Xingcheng Yao and Zhejun Jiang and Guokun Lai and Yulun Du and Yidao Qin and Weixin Xu and Enzhe Lu and Junjie Yan and Yanru Chen and Huabin Zheng and Yibo Liu and Shaowei Liu and Bohong Yin and Weiran He and Han Zhu and Yuzhi Wang and Jianzhou Wang and Mengnan Dong and Zheng Zhang and Yongsheng Kang and Hao Zhang and Xinran Xu and Yutao Zhang and Yuxin Wu and Xinyu Zhou and Zhilin Yang},
year = {2025},
eprint = {2502.16982},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2502.16982}
}
Muon made its name in a competition.
The NanoGPT speedrun is a public competition Keller Jordan maintains on GitHub. The task is to train a language model on 8 H100s to a loss of 3.28 on the FineWeb validation set in the shortest time; 3.28 is the score Andrej Karpathy reached when reproducing GPT-2 small. On October 15, 2024, Jordan switched the optimizer from AdamW to Muon, his own design, making training 35% faster in one go and setting a new record. Over the next two months, 7 researchers set 12 more records, every one of them still using Muon.
In December 2024, Jordan wrote a blog post explaining Muon's design. It ends with three questions he couldn't answer himself. Will it scale beyond 20 billion parameters and 1 trillion tokens? Can the Newton–Schulz iterations be distributed across large GPU clusters? Does it only work for pretraining, and fall short in fine-tuning or reinforcement learning?
Two months later, Moonshot AI, the company behind Kimi, published this technical report, answering the first two questions and part of the third. They used Muon to train an MoE model with 3 billion activated and 16 billion total parameters from scratch on 5.7 trillion tokens, and named it Moonlight.
Muon itself has no paper, so this note first lays out Muon from Jordan's blog post, then looks at the two things Moonshot AI added to scale it up, and finally at how the claim of “about 2× compute efficiency” held up when others checked it.
Defining the problem
What Muon is
Jordan's definition is just four lines (Algorithm 2 in the blog post):
The first two lines are just SGD with momentum. The difference is in the third line, where Muon first “orthogonalizes” the momentum matrix and then uses the result for the update. If has the singular value decomposition , orthogonalizing it means replacing it with , keeping the directions (, ) and setting every singular value to 1. NewtonSchulz5 is a routine that approximates this with five steps of matrix multiplication; we'll take it apart later.
This operation only makes sense for matrices, so Muon is used only on the 2D weights of hidden layers, such as the attention projections and feed-forward layers in a Transformer. Scalar and vector parameters (LayerNorm gains, biases), as well as the embedding layer and the final classification head, even though those two are 2D, go to AdamW. Convolution kernels have their last three dimensions flattened into a matrix.
In practice it uses Nesterov-style momentum, feeding into the orthogonalization, with a momentum coefficient of 0.95. The blog post also records an experiment by Vlado Boza showing that orthogonalizing the Q, K and V projections separately works better than treating them as one combined matrix.
Why orthogonalize
Jordan's empirical observation is that the updates SGD momentum and Adam produce for a Transformer's 2D weights both have very high condition numbers and are nearly low-rank. A few singular directions dominate the whole update, and the components along the other directions are negligibly small. His hypothesis is that those small directions, small as they are, still matter for learning, and that orthogonalization, by making every direction equally large, effectively amplifies these “rare directions.”
Another explanation comes from Jeremy Bernstein and Laker Newhouse's 2024 Old Optimizer, New Norm. They view an optimizer as taking the steepest step along the gradient under some norm constraint. Constrain the size of each coordinate (the norm), and the steepest step is the sign of the gradient; constrain the spectral norm of the whole matrix (its largest singular value), and the steepest step is . Adam with its EMA switched off reduces to the former, and Shampoo with its accumulation switched off is exactly the latter.
A weight matrix is a linear operator acting on hidden vectors, so it makes more sense to measure how much it has been changed with the spectral norm. Moonshot AI adopts this view in the paper too, arguing that Muon's norm constraint is more reasonable than Adam's.
What happens at scale
Muon shines on models of 1.5B and below. Jordan used Muon to train a 1.5B model to GPT-2 XL's HellaSwag level in 10 hours on 8×H100, versus 13.3 hours for AdamW. But these are all small models and short training runs.
As soon as Moonshot AI scaled up the model and the token count, they found Muon's advantage over AdamW shrinking. This paper sets out to find out why, and to make Muon usable out of the box for training runs of hundreds of billions to trillions of tokens, without extra tuning.
Solving the problem
Newton–Schulz: orthogonalization without SVD
An SVD gives directly, but it's too slow on GPUs. The coupled Newton iteration common in Shampoo implementations, Jordan found, needs float32 to stay stable, and is slow as well. Newton–Schulz iteration uses only matrix multiplications and runs stably in bfloat16.
Let's see what one iteration does to a matrix. With , one Newton–Schulz step is
and are left untouched; only each singular value goes through a quintic polynomial . Iterating times means applying times in a row.
So as long as two conditions hold, the iteration pushes the matrix toward .
- All singular values start within . Dividing the matrix by its Frobenius norm takes care of this, since orthogonalization doesn't care about overall scale.
- The coefficients are chosen so that applying repeatedly pushes every value in toward 1.
already converges. But what Jordan cares about is the number of steps. , so the larger is, the faster very small singular values get amplified. He also found that the final singular values needn't be exactly 1; anywhere between 0.7 and 1.3 leaves the training loss curve unaffected. So within that tolerance he made as large as possible, arriving at by gradient-based tuning, and five steps are enough.

The blue line is the conservative coefficient set. It converges to 1 cleanly, but a singular value has to be around 0.05 before it gets pushed up; smaller ones are still small after five steps, so the orthogonalization isn't finished. The orange line is Jordan's tuned set, which pushes even singular values as small as 0.003 to near 1, at the cost of results that oscillate between about 0.68 and 1.2. Muon uses this orange line, so strictly speaking what it computes is , with singular values that are only roughly equal.
The official implementation (KellerJordan/Muon) is just these lines:
def zeropower_via_newtonschulz5(G, steps: int):
a, b, c = (3.4445, -4.7750, 2.0315)
X = G.bfloat16()
if G.size(-2) > G.size(-1):
X = X.mT # so that X @ X.mT is the smaller side
X = X / (X.norm(dim=(-2, -1), keepdim=True) + 1e-7) # squeeze the singular values into [0, 1]
for _ in range(steps):
A = X @ X.mT
B = b * A + c * A @ A
X = a * X + B @ X # aX + b(XXᵀ)X + c(XXᵀ)²X
if G.size(-2) > G.size(-1):
X = X.mT
return X
The compute cost is modest too. For an matrix (), each iteration takes about floating-point operations. Jordan estimates the extra overhead at most , where is the number of iterations, the model dimension and the number of tokens per batch. For the NanoGPT speedrun, with and , the overhead is about 0.7%; for Llama 405B, with and around 16 million, it's about 0.5%.
Fix one: weight decay
Jordan's original Muon has no weight decay. When scaling up, Moonshot AI observed the RMS of the weights and of each layer's output growing steadily until it went beyond bfloat16's high-precision range. Their fix is to add the same decoupled weight decay that AdamW uses:
To check the effect, they trained an 800M-parameter model on 100 billion tokens, about 5 times the optimal token count from their scaling law. Muon without weight decay converges fastest at first, leading the version with weight decay by 0.023 around step 24,000; after that some weights grow too large, and by step 66,000 it has been overtaken by 0.017. In this overtrained regime, Muon with weight decay beats both the original Muon and AdamW.
Later independent evaluations back this up. Semenov et al. at EPFL compared the two versions on models from 124M to 720M; the original Muon degrades as models grow, while this paper's version (which they call D-Muon) stays stable in every setting.
Fix two: equal-sized updates for every matrix
The second problem hides in orthogonalization itself. Every singular value of is 1, so its squared Frobenius norm equals its rank . A full-rank matrix has rank , so the root mean square (RMS) of its entries is
This is Lemma 1 of the paper. The Muon update RMS measured during training is indeed close to this theoretical value.
Plug in a model with hidden dimension . The attention query projection is , with an update RMS of about ; the first layer of a conventional MLP is , with an update RMS of only about , half as large. Under the same learning rate, matrices of different shapes actually move different distances at each step.
The paper points out two situations where this goes wrong. Matrices with a large (such as a big dense MLP) get updates too small to learn much; matrices with a small (such as treating each KV head as a separate parameter in GQA or MLA) get updates that are too large, making training unstable.
The fix is to multiply each matrix's update by , canceling out Lemma 1. Moonshot AI takes it one step further. Muon handles only matrices, while the embedding layer, the classification head and RMSNorm still go to AdamW, and ideally the two optimizers share one learning rate and one weight decay. In their experience, AdamW's update RMS is about 0.2 to 0.4, so they scale Muon's update into that range as well:
With this, every matrix's update RMS is 0.2, and Muon can reuse the learning rate and weight decay already tuned for AdamW. This is the most practical formula in the paper. In Moonlight's open-source example, it's a single line, adjusted_ratio = 0.2 * math.sqrt(max(A, B)), used to scale up the learning rate on the Muon term; the weight-decay term is still multiplied by the original lr.
The 0.2 has experimental backing. Training a small model for 2,000 steps (about 2 billion tokens) with Muon's update RMS set to 0.05, 0.1, 0.2, 0.4 and 0.8 gives validation losses of 3.503, 3.374, 3.325, 3.314 and 3.543, against 3.679 for AdamW. 0.2 and 0.4 are about equally good, and the authors chose 0.2.
Another experiment compares three ways of controlling the RMS. To amplify the differences in shape, they replace the 800M model's SwiGLU with a two-layer MLP and evaluate at 4B tokens:
| Approach | Validation loss | MLP weight RMS |
|---|---|---|
| Multiply everything by | 2.812 | 0.0252 |
| Normalize the output directly to RMS 0.2 | 2.789 | 0.0501 |
| Multiply by | 2.789 | 0.0489 |
Compiled from Table 1 of the paper.
The first approach, which looks only at the hidden dimension , gives the MLP half the update of the other two (), and its loss is higher. The other two are equally good, and the paper chose the cheaper third one.
Other hyperparameters stay as they were. Ten Newton–Schulz steps orthogonalize more precisely but don't improve training results, so 5 steps are used; tuning the momentum gave no consistent gain, so it stays at 0.95.
Distributed Muon
Large-scale training usually uses ZeRO-1 to shard the optimizer state across data-parallel (DP) ranks, with each rank updating only its own shard of the parameters. That's fine for AdamW, which works elementwise; Muon, though, needs the full matrix to orthogonalize.
The paper's Distributed Muon adds two steps on top of ZeRO-1:
- Reduce-scatter the gradients to the ranks as usual, and update the local momentum shard;
- Added: within the DP group, gather the updated gradient shards into the full matrix;
- Added: run Newton–Schulz on the full matrix, keep only the part of the result that belongs to the local shard, and apply the update;
- All-gather the parameters as usual.
On memory, Muon keeps one momentum buffer and AdamW two, so its optimizer-state memory is half that of distributed AdamW. On communication, the extra gather involves only the local shard and runs in bfloat16, so total communication is 1 to 1.25 times that of distributed AdamW. On latency, the optimizer step usually takes only 1% to 3% of the forward-plus-backward time, and the gather can overlap with computation. The authors say they see no noticeable latency difference on their cluster.
Discussion
Scaling law: about 2×
For fairness, the authors first ran a scaling-law study for AdamW. Under several compute budgets, they varied model size and token count to find the best combinations, then grid-searched the learning rate and batch size, repeating for two or three rounds. The result is 5 dense Llama-architecture models from 399M to 1.5B (excluding embeddings), with 8.9B to 38.9B tokens and learning rates from 9.5e-4 to 8.3e-4. Since Muon's update RMS is already matched, it reuses AdamW's best hyperparameters directly.

The two fits are for Muon and for AdamW. The paper concludes that in the compute-optimal setting, Muon needs only about 52% of the training compute to reach AdamW's loss, which is about 2× the compute efficiency.
Working it out yourself from these two formulas, the compute ratio at equal loss is about 2.3× at 0.1 PFLOP/s-days, about 2.0× at 4 PFLOP/s-days near the largest experimental point, and down to about 1.9× extrapolated to 10 PFLOP/s-days. AdamW's exponent is slightly larger, so the two lines slowly converge. The ratio shrinking as compute grows is something we computed from the paper's fits; the paper doesn't discuss it.
Two more conditions of this experiment are worth remembering. The token counts in the table are about 22 to 26 times the parameter counts, close to the Chinchilla-optimal ratio (20 times), so only this one data regime was tested; and AdamW's learning rates were chosen within this scaling-law procedure, falling between 8e-4 and 1e-3. Both points were raised again later.
Moonlight: swapping only the optimizer
The production model uses the DeepSeek-V3-Small architecture, with 2.24B activated and 15.29B total parameters (3B/16B including embeddings), plus a few small changes such as removing multi-token prediction. The maximum context is 8K, and weight decay is 0.1 throughout. Training has three stages:
- First 33B tokens: the learning rate rises linearly to 4.2e-4 within 2,000 steps, with a batch of 2,048 sequences;
- 33B to 5.2T: cosine decay from 4.2e-4 to 4.2e-5, with the batch doubled to 4,096 sequences after 200B tokens;
- Final 500B tokens: the learning rate first rises to 1e-4, then decays linearly to 0, and the data switches to the highest-quality mix, mainly math, code and reasoning.
The cleanest comparison is Moonlight-A, with identical training settings except that the optimizer is switched back to AdamW. Both are compared at intermediate checkpoints at 1.2T tokens:
| Benchmark | AdamW | Muon |
|---|---|---|
| MMLU | 60.2 | 60.4 |
| MMLU-pro | 26.8 | 28.1 |
| BBH | 45.3 | 43.2 |
| HumanEval | 29.3 | 37.2 |
| MBPP | 49.2 | 52.9 |
| GSM8K | 43.8 | 45.0 |
| MATH | 16.1 | 19.8 |
Excerpted from Table 4 of the paper; both are intermediate checkpoints at 1.2T tokens, before the learning rate has finished decaying.
MMLU is nearly the same, and on BBH AdamW is actually higher. The biggest gaps are in code and math, with HumanEval up 7.9 points. The authors also point out that Muon does especially well on math and code, leaving the reason to future work.
After all 5.7T tokens, Moonlight's MMLU is 70.0; DeepSeek-V2-Lite, at the same parameter scale and also trained on 5.7T tokens but with AdamW, gets 58.3, and Qwen2.5-3B, trained on 18T tokens, gets 65.6. These come from different teams with different data, so all they show is that Moonlight sits at the frontier of the trade-off between training compute and performance; they can't serve as a controlled comparison of optimizers.
Training went smoothly, with no spikes in loss or gradient norm. The appendix has two details. First, the maximum attention logit in some layers climbed past 100 early in training, and the authors state that AdamW “demonstrated healthier behavior in controlling this metric”; logits above 100 made up only about one in ten thousand, and they came down gradually later on. Second, the RMSNorm gain parameters also need weight decay, or the RMS of each layer's output grows too large.
More spread-out updates, and weights too
The intuition that Muon makes weights learn along more directions can be measured. The authors compute the SVD entropy of each weight matrix:
The more even the singular values, the closer the entropy is to 1; when a few singular values dominate, the entropy is low. Throughout the 1.2T-token pretraining, across six groups of weights including attention, experts, shared experts, the router and the first dense layer, Muon's average SVD entropy is higher than AdamW's at every checkpoint, and over 90% of the weight matrices also have higher entropy under Muon. The largest gap is in the MoE router, from which the authors speculate that MoE models benefit more from Muon.
Switching optimizers for fine-tuning
This is a partial answer to Jordan's third question. The authors took the two 1.2T-token checkpoints and fine-tuned each with both Muon and AdamW on Tülu 3's SFT data for two epochs:
| Pretraining → SFT | MMLU | HumanEval | GSM8K |
|---|---|---|---|
| Muon → Muon | 55.7 | 57.3 | 68.0 |
| AdamW → Muon | 55.3 | 53.7 | 62.1 |
| Muon → AdamW | 50.2 | 52.4 | 64.9 |
| AdamW → AdamW | 52.0 | 53.1 | 64.6 |
Excerpted from Table 6 of the paper.
Using Muon all the way through works best. But once pretraining and fine-tuning use different optimizers, Muon has no clear advantage, and Muon pretraining followed by AdamW fine-tuning even has the lowest MMLU. Fine-tuning the public Qwen2.5-7B base model with Adam and with Muon gives similar results, with Muon's GSM8K actually 4 points lower (89.8 vs 85.8). The authors conclude that Muon's benefit lies mainly in pretraining, and they list this mismatch between pretraining and fine-tuning optimizers as an open problem. They also note that most existing pretrained checkpoints were trained with AdamW, and the mismatch makes it hard for them to benefit from Muon directly.
Later: Kimi K2's QK-Clip
The rising attention logits caused real trouble on larger models. The Kimi K2 technical report of July 2025 says that when they trained an MoE with 9B activated and 53B total parameters using vanilla Muon, the maximum attention logit quickly exceeded 1,000; logits of that magnitude usually bring obvious loss spikes, or even divergence. In their experiments, this problem was more common with Muon than with AdamW. Their hypothesis is that because Muon's updates have equal singular values and full effective rank, they more readily make the spectral norms of and grow together.
The fix is QK-Clip. After each update, look at each attention head's maximum logit on the current batch; if it exceeds a threshold , shrink that head's and together, with the scaling factor split evenly between the query and key sides (each multiplied by ; the RoPE part of MLA has its own rule). It changes only the weights, leaving the current step's forward and backward computation alone, and touches only the heads that exceed the limit.
Muon with weight decay and RMS matching (the formula from this paper), plus QK-Clip, is collectively called MuonClip. Kimi K2 used it to train a model with 1 trillion total and 32 billion activated parameters on 15.5 trillion tokens, with , without a single loss spike. QK-Clip was active only during the first 70k steps, during which 12.7% of the attention heads triggered it at some point; after that every head's maximum logit fell below 100 on its own, and the safeguard never kicked in again.
Moonshot AI wasn't the only one to adopt Muon afterward. Zhipu's GLM-4.5 (August 2025) uses Muon for all parameters except word embeddings, biases and RMSNorm weights, with settings taken straight from this paper: 5 iterations, momentum 0.95 and update RMS scaled to 0.2. DeepSeek-V4 also switched most of its parameters to Muon; its attention already applies RMSNorm to the query and the KV, so its logits don't blow up and it doesn't need QK-Clip.
What became of “2×”
The 2× figure has been re-examined several times.
Wen et al. at Stanford (September 2025) ran a full hyperparameter sweep for every optimizer before comparing token efficiency. They measured Muon at about 1.3 to 1.4 times AdamW from 130M to 520M, dropping to about 1.1 times at 1.2B. In their appendix, they single out two conditions of this paper. First, AdamW's learning rate was on the low side, 8e-4 to 9e-4 for 400M to 1.5B, while the best values they tuned were higher; second, the comparison between Moonlight and Moonlight-A was taken at intermediate checkpoints before the learning rate had finished decaying, and their experiments show Muon often leads by a wide margin early on, only to be caught during the decay phase.
An independent study by Essential AI (May 2025, 100M to 4B) concludes that Muon needs 10% to 15% fewer tokens than AdamW to reach the same loss, and that its advantage holds, or even grows, when the batch is scaled far beyond the critical batch size. NVIDIA, in July 2026, with MoEs from 30B to 72B and 25 million or more tokens per batch, also saw AdamW degrade at large batch sizes while Muon held up.
Taken together, Muon's improvement is real, but against a well-tuned AdamW the gap in token efficiency is much smaller than 2×; its clearer advantage is in large-batch training. The paper's 2× came from a scaling law near 1× Chinchilla with a low AdamW learning rate, and the Moonlight vs Moonlight-A comparison was taken at intermediate checkpoints before the learning rate had finished decaying.
Conclusion
Moonshot AI gave Muon decoupled weight decay so that the RMS of the weights and outputs stops growing, then multiplied the update by so that every matrix's update RMS matches AdamW's 0.2. With a ZeRO-1-style distributed implementation on top, Muon ran a 5.7-trillion-token MoE pretraining to completion, reusing AdamW's hyperparameters directly. The roughly 2× compute efficiency has to be read with its conditions; later studies with well-tuned baselines measured 1.1 to 1.4×.
If you want to switch to Muon, leave the embedding layer, classification head, LayerNorm and biases to AdamW, and give the other 2D weights to Muon. Check which scaling your implementation uses; the Moonlight style can take AdamW's learning rate directly, while Jordan's style needs a fresh sweep. If what you have is a model pretrained with AdamW, try Muon fine-tuning at a small scale first.
At the end of his blog post, Jordan wrote that he didn't know whether Muon could scale to 20 billion parameters and 1 trillion tokens. Seven months later, Kimi K2 used it to train a 1-trillion-parameter model on 15.5 trillion tokens.