Training Language Models to Self-Correct via Reinforcement Learning

By Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D. Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs and 8 more. International Conference on Learning Representations, 2024

429 citations recorded by the source corpus.

Self-correction is a highly desirable capability of large language models (LLMs), yet it has consistently been found to be largely ineffective in modern LLMs. Current methods for training self-correction typically depend on either multiple models, a more advanced model, or additional forms of supervision. To address these shortcomings, we develop a multi-turn online reinforcement learning (RL) approach, SCoRe, that significantly improves an LLM's self-correction ability using entirely self-generated data. To build SCoRe, we first show that variants of supervised fine-tuning (SFT) on offline model-generated correction traces are often insufficient for instilling self-correction behavior. In particular, we observe that training via SFT falls prey to either a distribution mismatch between mistakes made by the data-collection policy and the model's own responses, or to behavior collapse, where learning implicitly prefers only a certain mode of correction behavior that is often not effective at self-correction on test problems. SCoRe addresses these challenges by training under the model's own distribution of self-generated correction traces and using appropriate regularization to steer the learning process into learning a self-correction behavior that is effective at test time as opposed to fitting high-reward responses for a given prompt. This regularization process includes an initial phase of multi-turn RL on a base model to generate a policy initialization that is less susceptible to collapse, followed by using a reward bonus to amplify self-correction. With Gemini 1.0 Pro and 1.5 Flash models, we find that SCoRe achieves state-of-the-art self-correction performance, improving the base models' self-correction by 15.6% and 9.1% respectively on MATH and HumanEval.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c4ff8bc44d88cd267baf18ac5d3a3a1fe86d08eb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhenrui Yue
2. Xingyu Chen
3. Hamed Zamani
4. Jiawei Han
5. Bowen Jin
6. Hansi Zeng
7. Dong Wang

Citations in the corpus

• rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (2025, neutral): “The community has actively explored various approaches, including self-correction [Huang et al., 2023, Kumar et al., 2024], self-reflection [Renze and Guven, 2024, Shinn et al., 2024], to explicitly train or prompt LLMs to develop such capability.”
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025, neutral): “…various approaches, including process-based reward models (Lightman et al., 2023; Uesato et al., 2022; Wang et al., 2023), reinforcement learning (Kumar et al., 2024), and search algorithms such as Monte Carlo Tree Search and Beam Search (Feng et al., 2024; Trinh et al., 2024; Xin et al., 2024).”
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs (2025, neutral): “…achieved through various approaches such as 1) in-context examples (Gandhi et al., 2023), 2) finetuning on linearized search traces (Gandhi et al., 2024; Lehnert et al., 2024), and 3) training on self-correction examples (Ye et al., 2024; Qu et al., 2025; Kumar et al., 2024; Hwang et al., 2024).”
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (2025, positive): “Reinforcement Learning (RL) (Sutton et al., 1999; Kaelbling et al., 1996) has emerged as a potent paradigm for enhancing the reasoning capabilities of LLMs (Guo et al., 2025; Hou et al., 2025; Xie et al., 2025; Kumar et al., 2024).”
• Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs (2024, positive): “This has been expanded with methods like debating, self-correction, self-critique, and plan-and-solve (Liang et al., 2024; Du et al., 2024; Xiong et al., 2023; Kumar et al., 2024; Kamoi et al., 2024; Ke et al., 2023; Lin et al., 2024; Yu et al., 2024; Wang et al., 2023a).”