CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

By Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, Steven C. H. Hoi. 2022

502 citations recorded by the source corpus.

Program synthesis or code generation aims to generate a program that satisfies a problem specification. Recent approaches using large-scale pretrained language models (LMs) have shown promising results, yet they have some critical limitations. In particular, they often follow a standard supervised fine-tuning procedure to train a code generation model only from the pairs of natural-language problem descriptions and ground-truth programs. Such paradigm largely ignores some important but potentially useful signals in the problem specification such as unit tests, which thus often results in poor performance when solving complex unseen coding tasks. To address the limitations, we propose"CodeRL", a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning (RL). Specifically, during training, we treat the code-generating LM as an actor network, and introduce a critic network that is trained to predict the functional correctness of generated programs and provide dense feedback signals to the actor. During inference, we introduce a new generation procedure with a critical sampling strategy that allows a model to automatically regenerate programs based on feedback from example unit tests and critic scores. For the model backbones, we extended the encoder-decoder architecture of CodeT5 with enhanced learning objectives, larger model sizes, and better pretraining data. Our method not only achieves new SOTA results on the challenging APPS benchmark, but also shows strong zero-shot transfer capability with new SOTA results on the simpler MBPP benchmark.

Subfields: Information Systems, Software, Large Language Models, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Large Language Models

In Reinforcement Learning

Citations in the corpus

Self-Refine: Iterative Refinement with Self-Feedback (2023, background): “Rather than having an explicit refinement module, an alternative way to incorporate feedback is by optimizing a scalar reward function, e.g. with reinforcement learning (e.g., Stiennon et al. (2020); Lu et al. (2022); Le et al. (2022a)).”
Code Llama: Open Foundation Models for Code (2023, background): “…filter, cluster, or improve the correctness of programs when few candidates must be submitted (Li et al., 2022; Chen et al., 2023a; Le et al., 2022; Zhang et al., 2023), or unit tests them within a reinforcement learning objective to enrich the training signal (Le et al., 2022; Liu et al., 2023a).”
CodeT: Code Generation with Generated Tests (2022, background): “Moreover, some work has been proposed to leverage the execution information (Shi et al., 2022; Li et al., 2022b; Le et al., 2022; Lahiri et al., 2022).”
Planning with Large Language Models for Code Generation (2023, methodology): “In the training phase, Le et al. (2022); Xu et al. (2019b) use an RL objective that optimizes the correctness of the generated programs instead of optimizing their similarity to the reference solutions.”
Cognitive Architectures for Language Agents (2023, background): “, 2023) as well as general code execution (Yang et al., 2023a; Le et al., 2022; Ni et al., 2023).”
RLTF: Reinforcement Learning from Unit Test Feedback (2023, methodology): “CodeRL (Le et al., 2022) PPOCoder (Shojaee et al., 2023) RLTF errors based on their reported information and location, penalizing the specific erroneous parts of the code accordingly.”
Generating Sequences by Learning to Self-Correct (2022, methodology): “Self-Correction relates to work on program synthesis (Fu et al., 2019; Balog et al., 2020; Gupta et al., 2020; Le et al., 2022) and repair (Gupta et al.”
DAPO: An Open-Source LLM Reinforcement Learning System at Scale (2025, unknown): “This is proved to be an effective approach to activating the base model’s reasoning capability, as shown in multiple domains such as automated theorem proving [30–33], computer programming [34–37], and mathematics competition [2].”
Demystifying GPT Self-Repair for Code Generation (2023, methodology): “The use of large language models for program synthesis has been studied extensively in the literature [2, 5, 9, 11, 18, 20, 21, 25, 32].”
Language Models Can Teach Themselves to Program Better (2022, methodology): “Finally, the self-play pipeline could be combined with the popular topic of search strategies for code generation [e.g., Li et al., 2022; Le et al., 2022; Ellis et al., 2019].”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “, 2023) SL Program Executor Scalar ✗ (rerank) High ✗ CodeRL (Le et al., 2022) RL Trained critic model Scalar ✗ (post-hoc) High ✗ Self-critique (Saunders et al.”
Textually Pretrained Speech Language Models (2023, background): “Such LMs often serve as a foundational model to be fine-tuned on other tasks such as text classification [Howard and Ruder, 2018], textual instructions [Ouyang et al., 2022], or code generation [Le et al., 2022, Nijkamp et al., 2022].”
Teaching Large Language Models to Self-Debug (2023, background): “Program repair is an area of research concerned with fixing bugs in code, where several neural network models have been developed to handle different types of bugs (Gupta et al., 2017; Yasunaga & Liang, 2021; Gupta et al., 2020; Wang et al., 2018; Le et al., 2022).”
Reflexion: language agents with verbal reinforcement learning (2023, unknown): “CodeRL [12] sets the problem in an RL framework using an actor-critic setup to debug programs given feedback from an execution environment.”
Mastering Diverse Domains through World Models (2023, background): “It is also a key component for improving large language models beyond what is demonstrated in their pretraining data 3,4 .”
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, unknown): “, 2022], we explore to combine span denoising with CLM tasks to improve the model with better causal generation capability [Le et al., 2022].”
Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, methodology): “AutoGPT [28] is a popular software tool that automates NLP tasks by decomposing a high-level goal into multiple subgoals and executing them in a ReAct-style loop.”
Agentic Reinforcement Learning for Real-World Code Repair (2025, unknown): “Recent research has extended their reach into more realistic environments: CodeRL (Le et al., 2022a) integrated reinforcement learning (RL) to improve correctness in competitive programming, CoRNStack (Suresh et al., 2024) advanced large-scale code retrieval and reranking, and surveys (e.g., Wang…”