Understanding R1-Zero-Like Training: A Critical Perspective

By Zi-Yan Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin. arXiv.org, 2025

1,176 citations recorded by the source corpus.

DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we critically examine R1-Zero-like training by analyzing its two core components: base models and RL. We investigate a wide range of base models, including DeepSeek-V3-Base, to understand how pretraining characteristics influence RL performance. Our analysis reveals that DeepSeek-V3-Base already exhibit''Aha moment'', while Qwen2.5 base models demonstrate strong reasoning capabilities even without prompt templates, suggesting potential pretraining biases. Additionally, we identify an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training. To address this, we introduce Dr. GRPO, an unbiased optimization method that improves token efficiency while maintaining reasoning performance. Leveraging these insights, we present a minimalist R1-Zero recipe that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a new state-of-the-art. Our code is available at https://github.com/sail-sg/understand-r1-zero.

View Paper: https://www.semanticscholar.org/paper/de23d38bc2604dcf334dcc46aff217eb6bcd1fe1

Citations in the corpus

• Group-in-Group Policy Optimization for LLM Agent Training (2025, unknown): “These methods, such as RLOO [15, 16], GRPO [17], Dr. GRPO [18], DAPO [19], and CPPO [56], avoid introducing extra value functions by leveraging a group of samples from the same query and estimate the advantages accordingly.”
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025, unknown): “5-7B-Base and trained using the Oat-Zero framework, for further comparison (Liu et al., 2025b).”
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model (2025, unknown): “• All assumptions should be clearly stated or referenced in the statement of any theorems.”
• DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical Reasoning (2025, unknown): “Thus, Graph-Cut SMI provides a principled measure of o i ’s relative redundancy (high value) or diversity (low value) within the group.”