Muon is Scalable for LLM Training

By Jingyuan Liu, Jianling Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan and 18 more. arXiv.org, 2025

305 citations recorded by the source corpus.

Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale. These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon achieves $\sim\!2\times$ computational efficiency compared to AdamW with compute optimal training. Based on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models. We open-source our distributed Muon implementation that is memory optimal and communication efficient. We also release the pretrained, instruction-tuned, and intermediate checkpoints to support future research.

Subfields: Aerospace Engineering, Biomedical Engineering, Nuclear and High Energy Physics

View Paper: https://www.semanticscholar.org/paper/1d239507d99278b79b85c04eab7b59fd7e3e3216

Citations in the corpus

• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (2025, unknown): “We employed the Muon optimizer [21; 24] for all parameters except word embedding, bias, and weights for RMSNorm.”
• Kimi K2: Open Agentic Intelligence (2025, unknown): “Specifically, we employ the token-efficient Muon optimizer [34, 47] and mitigate its training instabilities through the introduction of QK-Clip.”
• Eliciting Latent Predictions from Transformers with the Tuned Lens (2023, unknown): “While they’re not officially the same model suite, they’re more consistent than the OPT models. bias (Figure 3), and much lower perplexity than the logit lens across the board (Figure 5, Appendix A).”