DAPO: An Open-Source LLM Reinforcement Learning System at Scale

By Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu and 25 more. Neural Information Processing Systems, 2025

2,219 citations recorded by the source corpus.

Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the community still struggles to reproduce their RL training results. We propose the $\textbf{D}$ecoupled Clip and $\textbf{D}$ynamic s$\textbf{A}$mpling $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{DAPO}$) algorithm, and fully open-source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RL a success. In addition, we open-source our training code, which is built on the verl framework, along with a carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research in large-scale LLM RL.

View Paper: https://www.semanticscholar.org/paper/dd4cfde3e135f799a9a71b4f57e13a29de89f7e3

Citations in the corpus

• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models (2025, unknown): “Therefore, most recent works do not include them as well (Cui et al., 2025; Hu et al., 2025; Liu et al., 2025; Yu et al., 2025).”
• Group-in-Group Policy Optimization for LLM Agent Training (2025, unknown): “These methods, such as RLOO [15, 16], GRPO [17], Dr. GRPO [18], DAPO [19], and CPPO [56], avoid introducing extra value functions by leveraging a group of samples from the same query and estimate the advantages accordingly.”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, unknown): “In contrast, despite the effectiveness of online RL algorithms [108, 109, 167, 184], they are often computationally expensive and time-consuming.”
• VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning (2025, unknown): “GRPO-Filter removes the SSR component from GRPO-SSR (similar to the dynamic filtering in DAPO [Yu et al., 2025], but don’t involve an online re-sampling), while GRPO further removes the filtering of examples with zero advantages.”
• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, unknown): “In multimodal RL, similar to [66], increasing the upper clipping bound of the importance sampling ratio also proved useful for improving both off-policy performance and preventing excessive entropy collapse.”
• ToolRL: Reward is All Tool Learning Needs (2025, unknown): “Extensions like dynamic sampling policy optimization (DAPO) (Yu et al., 2025) and the more recent value-based augmented proximal policy optimization (VAPO) (Yuan et al., 2025) further improve training stability and efficiency.”