Qwen2.5 Technical Report
By Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong and 33 more. arXiv.org, 2024
4,487 citations recorded by the source corpus.
In this report, we introduce Qwen2.5, a comprehensive series of large language models (LLMs) designed to meet diverse needs. Compared to previous iterations, Qwen 2.5 has been significantly improved during both the pre-training and post-training stages. In terms of pre-training, we have scaled the high-quality pre-training datasets from the previous 7 trillion tokens to 18 trillion tokens. This provides a strong foundation for common sense, expert knowledge, and reasoning capabilities. In terms of post-training, we implement intricate supervised finetuning with over 1 million samples, as well as multistage reinforcement learning. Post-training techniques enhance human preference, and notably improve long text generation, structural data analysis, and instruction following. To handle diverse and varied use cases effectively, we present Qwen2.5 LLM series in rich sizes. Open-weight offerings include base and instruction-tuned models, with quantized versions available. In addition, for hosted solutions, the proprietary models currently include two mixture-of-experts (MoE) variants: Qwen2.5-Turbo and Qwen2.5-Plus, both available from Alibaba Cloud Model Studio. Qwen2.5 has demonstrated top-tier performance on a wide range of benchmarks evaluating language understanding, reasoning, mathematics, coding, human preference alignment, etc. Specifically, the open-weight flagship Qwen2.5-72B-Instruct outperforms a number of open and proprietary models and demonstrates competitive performance to the state-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5 times larger. Qwen2.5-Turbo and Qwen2.5-Plus offer superior cost-effectiveness while performing competitively against GPT-4o-mini and GPT-4o respectively. Additionally, as the foundation, Qwen2.5 models have been instrumental in training specialized models such as Qwen2.5-Math, Qwen2.5-Coder, QwQ, and multimodal models.
Subfields: Artificial Intelligence, Health Informatics
View Paper: https://www.semanticscholar.org/paper/88aa6b1f37d1fd8e0a40499ce9bb87873f03aaa8
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- ReTool: Reinforcement Learning for Strategic Tool Use in LLMs (2025, positive): “Recent advancements in large language models (LLMs) [3, 4, 9, 12, 19, 20, 25, 27, 29, 31] indicate significant progress toward cognitive abilities similar to human metacognition through Chain-of-Thought (CoT) prompting.”
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale (2025, neutral): “5-32B [12] as the pretrained model for RL.”
- Process Reinforcement through Implicit Rewards (2025, neutral): “5-32B-Base (Yang et al., 2024a), skipping the SFT phase.”
- Group-in-Group Policy Optimization for LLM Agent Training (2025, neutral): “5-7B-Instruct [3] as our base models.”
- The Lessons of Developing Process Reward Models in Mathematical Reasoning (2025, neutral): “5-72B-Instruct (Yang et al., 2024b) and QwQ-32B-Preview (Qwen, 2024).”
- Understanding R1-Zero-Like Training: A Critical Perspective (2025, neutral): “5 family (Yang et al., 2024a;b), Llama-3.”
- Large Language Diffusion Models (2025, positive): “Following standard LLM [25, 26] evaluation practices, we assess LLaDA across four dimensions: General ability: MMLU [110], BBH [111], ARC-C [112], Hellaswag [113], TruthfulQA [114], WinoGrande [115] and PIQA [116].”
- Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, neutral): “We defer the detailed discussion on these extensions to Appendix A.”
- Qwen3 Technical Report (2025, positive): “5-MoE (Yang et al., 2024b) and implement fine-grained expert segmentation (Dai et al., 2024).”
- SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild (2025, neutral): “5 (0.5B, 1.5B, 7B, 14B, 32B) (Yang et al., 2024a).”
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, neutral): “We compare the scores with the latest open-source models - Qwen 2.5 [YYZ + 24], Llama-3.”
- Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (2025, neutral): “Evolving from foundational pretrained models (e.g., LLaMA [32,86]) trained with next-token prediction [25], these models typically leverage Chain-of-Thought (CoT) [92] BaseModel 1: The pipeline of developing efficient reasoning for LLMs.”
- An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning (2023, neutral): “5-Inst [10] with different scales from 3 B to 14 B, whose results are shown in Fig.”
- Muon is Scalable for LLM Training (2025, neutral): “5-7B (Yang et al. 2024).”
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025, neutral): “We compare the pass@ k curves of base and zero-RL models on benchmarks of varying difficulty: GSM8K (Cobbe et al., 2021), MATH500 (Hendrycks et al., 2021), Minerva (Lewkowycz et al., 2022), Olympiad (He et al., 2024), AIME24, and AMC23.”
- Emerging Properties in Unified Multimodal Pretraining (2025, neutral): “5-72B [93] to generate additional query-guidance pairs and corresponding detailed prompts, which are then passed to FLUX .1-dev [35] to produce target images.”
- DanceGRPO: Unleashing GRPO on Visual Generation (2025, neutral): “Large Language Models (LLMs) [42–46] are typically aligned with Rein-forcement Learning from Human Feedback (RLHF) [45–50].”