SmolLM2: When Smol Goes Big - Data-Centric Training of a Small Language Model
By Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart'in Bl'azquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydl'ivcek, Agust'in Piqueres Lajar'in, Vaibhav Srivastav and 12 more. arXiv.org, 2025
299 citations recorded by the source corpus.
While large language models have facilitated breakthroughs in many applications of artificial intelligence, their inherent largeness makes them computationally expensive and challenging to deploy in resource-constrained settings. In this paper, we document the development of SmolLM2, a state-of-the-art"small"(1.7 billion parameter) language model (LM). To attain strong performance, we overtrain SmolLM2 on ~11 trillion tokens of data using a multi-stage training process that mixes web text with specialized math, code, and instruction-following data. We additionally introduce new specialized datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing datasets to be problematically small or low-quality. To inform our design decisions, we perform both small-scale ablations as well as a manual refinement process that updates the dataset mixing rates at each stage based on the performance at the previous stage. Ultimately, we demonstrate that SmolLM2 outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To facilitate future research on LM development as well as applications of small LMs, we release both SmolLM2 as well as all of the datasets we prepared in the course of this project.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/54145fcb714653b1143bef312fdac821cccbfb62
Citations in the corpus
- Understanding R1-Zero-Like Training: A Critical Perspective (2025, unknown): “We implement our algorithm using Oat (Liu et al., 2025a), a modular, research-friendly and efficient LLM RL framework.”
- SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics (2025, unknown): “SmolVLM-2 relies on SigLIP (Zhai et al., 2023) to encode visual features for SmolLM2 language decoder (Allal et al., 2025).”
- Zero-Shot Resume–Job Matching with LLMs via Structured Prompting and Semantic Embeddings (2025, unknown): “7B [25], Qwen3-1.”
- Seq vs Seq: An Open Suite of Paired Encoders and Decoders (2025, unknown): “Following previous work (Allal et al., 2025) the ARC score is the average of the easy and challenge sets.”