MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

By Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao and 15 more. arXiv.org, 2024

750 citations recorded by the source corpus.

The burgeoning interest in developing Large Language Models (LLMs) with up to trillion parameters has been met with concerns regarding resource efficiency and practical expense, particularly given the immense cost of experimentation. This scenario underscores the importance of exploring the potential of Small Language Models (SLMs) as a resource-efficient alternative. In this context, we introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter variants, not only excel in their respective categories but also demonstrate capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach exhibits scalability in both model and data dimensions for future LLM research. Regarding model scaling, we employ extensive model wind tunnel experiments for stable and optimal scaling. For data scaling, we introduce a Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to continuous training and domain adaptation. We present an in-depth analysis of the intriguing training dynamics that occurred in the WSD LRS. With WSD LRS, we are now able to efficiently study data-model scaling law without extensive retraining experiments on both axes of model and data, from which we derive the much higher compute optimal data-model ratio than Chinchilla Optimal. Additionally, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE and MiniCPM-128K, whose excellent performance further cementing MiniCPM's foundation in diverse SLM applications. MiniCPM models are available publicly at https://github.com/OpenBMB/MiniCPM .

View Paper: https://www.semanticscholar.org/paper/49873ee415619efd9e1e4c16f73ee066ff008c1f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Zhihong Shao
2. Daya Guo
3. Damai Dai
4. Zihan Wang
5. Huajian Xin
6. Peiyuan Zhang
7. Guangtao Zeng
8. Tianduo Wang

Citations in the corpus

• Kimi K2: Open Agentic Intelligence (2025, unknown): “We pre-trained the model with a 4,096-token context window using the MuonClip optimizer (Algorithm 1) and the WSD learning rate schedule [26], processing a total of 15.5T tokens.”
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024, background): “Finally, it is worth mentioning that certain prior studies (Hu et al., 2024) Inference Efficiency.”
• Large Language Diffusion Models (2025, unknown): “We adopted the Warmup-Stable-Decay [28] learning rate scheduler to monitor the training progress without interrupting continuous training.”
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (2025, unknown): “We used cosine decay schedule for learning rate, instead of warmup-stable-decay (WSD) schedule [17].”
• ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024, unknown): “These limitations have largely restricted most LLMs from being applied to more complex tasks.”
• TinyLlama: An Open-Source Small Language Model (2024, methodology): “Traditionally, this is achieved by modifying the learning rate, as seen in approaches like MiniCPM (Hu et al., 2024) and DeepSeek LLMs (Bi et al., 2024).”
• Emerging Properties in Unified Multimodal Pretraining (2025, unknown): “We adopt a constant learning rate for the PT, CT, and SFT stages so that we can easily scale the training data without restarting the training process [30].”
• Granite Embedding Multilingual R2 Models (2026, unknown): “We use a Warmup-Stable-Decay learning rate schedule (Hu et al., 2024), with a RoPE theta of 10,000.”
• ModernVBERT: Towards Smaller Visual Document Retrievers (2025, unknown): “We employ WSD scheduler (Hu et al., 2024b) with the first 5% of the training as warmup, the last 20% as decay and a maximum learning rate of 1e-4.”
• Granite Embedding R2 Models (2025, unknown): “We use a Warmup-Stable-Decay learning rate schedule (Hu et al., 2024), with peak learning rate of 8e-4 after an initial warmup on 3 billion tokens, and a RoPE theta of 10,000.”