![\"\"](\"https://github.com/MoonshotAI/Moonlight/raw/master/figures/banner.png\")
Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale. These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon achieves ∼ 2× computational efficiency compared to AdamW with compute optimal training.
\nBased on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models.
\nWe open-source our distributed Muon implementation that is memory optimal and communication efficient. We also release the pretrained, instruction-tuned, and intermediate checkpoints to support future research.
\nOur code is available at MoonshotAI/Moonlight.
\nOur work builds upon Muon while systematically identifying and resolving its limitations in large-scale training scenarios. Our technical contributions include:
\nAnalysis for Effective Scaling of Muon: Through extensive analysis, we identify that weight decay plays a crucial roles in Muon's scalability. Besides, we proposed to keep a consistent update root mean square (RMS) across different matrix and non-matrix parameters through parameter-wise update scale adjustments. Such adjustments significantly enhanced training stability.
\nEfficient Distributed Implementation: We develop a distributed version of Muon with ZeRO-1 style optimization, achieving optimal memory efficiency and reduced communication overhead while preserving the mathematical properties of the algorithm.
\nScaling Law Validation: We performed scaling law research that compares Muon with strong AdamW baselines, and showed the superior performance of Muon (see Figure 1). Based on the scaling law results, Muon achieves comparable performance to AdamW trained counterparts while requiring only approximately 52% of the training FLOPs.
\n![\"\"](\"https://github.com/MoonshotAI/Moonlight/blob/master/figures/scaling.png\")
Scaling up with Muon. (a) Scaling law experiments comparing Muon and Adam. Muon is 2 times more sample efficient than Adam. (b) The MMLU performance of our Moonlight model optimized with Muon and other comparable models. Moonlight advances the Pareto frontier of performance vs training FLOPs.
\nWe named our lightweight model trained with Muon "Moonlight". We compared Moonlight with SOTA public models at similar scale:
\n| \n | Benchmark (Metric) | \nLlama3.2-3B | \nQwen2.5-3B | \nDSV2-Lite | \nMoonlight | \n
|---|---|---|---|---|---|
| \n | Activated Param† | \n2.81B | \n2.77B | \n2.24B | \n2.24B | \n
| \n | Total Params† | \n2.81B | \n2.77B | \n15.29B | \n15.29B | \n
| \n | Training Tokens | \n9T | \n18T | \n5.7T | \n5.7T | \n
| \n | Optimizer | \nAdamW | \n* | \nAdamW | \nMuon | \n
| English | \nMMLU | \n54.75 | \n65.6 | \n58.3 | \n70.0 | \n
| \n | MMLU-pro | \n25.0 | \n34.6 | \n25.5 | \n42.4 | \n
| \n | BBH | \n46.8 | \n56.3 | \n44.1 | \n65.2 | \n
| \n | TriviaQA‡ | \n59.6 | \n51.1 | \n65.1 | \n66.3 | \n
| Code | \nHumanEval | \n28.0 | \n42.1 | \n29.9 | \n48.1 | \n
| \n | MBPP | \n48.7 | \n57.1 | \n43.2 | \n63.8 | \n
| Math | \nGSM8K | \n34.0 | \n79.1 | \n41.1 | \n77.4 | \n
| \n | MATH | \n8.5 | \n42.6 | \n17.1 | \n45.3 | \n
| \n | CMath | \n- | \n80.0 | \n58.4 | \n81.1 | \n
| Chinese | \nC-Eval | \n- | \n75.0 | \n60.3 | \n77.2 | \n
| \n | CMMLU | \n- | \n75.0 | \n64.3 | \n78.2 | \n
Qwen 2 & 2.5 reports didn't disclose their optimizer information. †The reported parameter counts exclude the embedding parameters. ‡We test all listed models with the full set of TriviaQA.
\n| Model | \n#Total Params | \n#Activated Params | \nContext Length | \nDownload Link | \n
|---|---|---|---|---|
| Moonlight | \n16B | \n3B | \n8K | \n🤗 Hugging Face | \n
| Moonlight-Instruct | \n16B | \n3B | \n8K | \n🤗 Hugging Face | \n
We introduce how to use our model at inference stage using transformers library. It is recommended to use python=3.10, torch>=2.1.0, and transformers=4.48.2 as the development environment.
\nFor our pretrained model (Moonlight):
\n\n\n = \n = \n = \n\n = \n = \n = \n = \n\nFor our instruct model (Moonlight-Instruct):
\n\n\n = \n = \n = \n\n = \n = \n = \n = \n\nMoonlight has the same architecture as DeepSeek-V3, which is supported by many popular inference engines, such as VLLM and SGLang. As a result, our model can also be easily deployed using these tools.
\n# train qwen-like dense model with muon\npython3 examples/toy_train.py --model qwen --optimizer muon --dataset openwebtext-100k --hidden_size 896 --lr 1e-3\n\n# train qwen-like dense model with adamw\npython3 examples/toy_train.py --model qwen --optimizer adamw --dataset openwebtext-100k --hidden_size 896 --lr 1e-3\nTo support ongoing research efforts, we will soon release our intermediate checkpoints. Coming soon...
\nIf you find Moonlight is useful or want to use in your projects, please kindly cite our paper:
\n@misc{liu2025muonscalablellmtraining,\n title={Muon is Scalable for LLM Training}, \n author={Jingyuan Liu and Jianlin Su and Xingcheng Yao and Zhejun Jiang and Guokun Lai and Yulun Du and Yidao Qin and Weixin Xu and Enzhe Lu and Junjie Yan and Yanru Chen and Huabin Zheng and Yibo Liu and Shaowei Liu and Bohong Yin and Weiran He and Han Zhu and Yuzhi Wang and Jianzhou Wang and Mengnan Dong and Zheng Zhang and Yongsheng Kang and Hao Zhang and Xinran Xu and Yutao Zhang and Yuxin Wu and Xinyu Zhou and Zhilin Yang},\n year={2025},\n eprint={2502.16982},\n archivePrefix={arXiv},\n primaryClass={cs.LG},\n url={https://arxiv.org/abs/2502.16982}, \n}\n