Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization

By Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, Yejin Choi. 2022

301 citations recorded by the source corpus.

We tackle the problem of aligning pre-trained large language models (LMs) with human preferences. If we view text generation as a sequential decision-making problem, reinforcement learning (RL) appears to be a natural conceptual framework. However, using RL for LM-based generation faces empirical challenges, including training instability due to the combinatorial action space, as well as a lack of open-source libraries and benchmarks customized for LM alignment. Thus, a question rises in the research community: is RL a practical paradigm for NLP? To help answer this, we first introduce an open-source modular library, RL4LMs (Reinforcement Learning for Language Models), for optimizing language generators with RL. The library consists of on-policy RL algorithms that can be used to train any encoder or encoder-decoder LM in the HuggingFace library (Wolf et al. 2020) with an arbitrary reward function. Next, we present the GRUE (General Reinforced-language Understanding Evaluation) benchmark, a set of 6 language generation tasks which are supervised not by target strings, but by reward functions which capture automated measures of human preference. GRUE is the first leaderboard-style evaluation of RL algorithms for NLP tasks. Finally, we introduce an easy-to-use, performant RL algorithm, NLPO (Natural Language Policy Optimization) that learns to effectively reduce the combinatorial action space in language generation. We show 1) that RL techniques are generally better than supervised methods at aligning LMs to human preferences; and 2) that NLPO exhibits greater stability and performance than previous policy gradient methods (e.g., PPO (Schulman et al. 2017)), based on both automatic and human evaluations.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, background): “reinforcement learning algorithms, commonly REINFORCE [41], proximal policy optimization (PPO; [34]), or variants [29].”
Augmented Language Models: a Survey (2023, methodology): “Optimization is done through policy gradient algorithms like REINFORCE (Williams, 1992), PPO and similar variants (Schulman et al., 2017; Ramamurthy et al., 2022).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “…been suggested as a better alternative (Alabdulkarim et al., 2021; Liu et al., 2021d; Ouyang et al., 2022; Stiennon et al., 2020; Lu et al., 2022; Ramamurthy et al., 2022) for training modern LMs, as RL models do not require carefully curated, balanced datasets and can instead learn from…”
RRHF: Rank Responses to Align Language Models with Human Feedback without tears (2023, methodology): “[20] use supervised fine-tuned models as the initial models, so we also have fine-tuned Alpaca-7B on our used dataset5 with chosen responses (i.”
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025, unknown): “First, the action space in language models is exponentially larger than that of Go or Atari games (Ramamurthy et al., 2023).”
Reinforcement Learning for Language Models (2023, background): “Recently, the authors of RL4LM [2] proposed a novel on-policy RL algorithm named Natural Language Policy Optimization (NLPO).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “Several techniques for aligning LLMs with human preferences obtain results competitive with RLHF by using supervised learning to complement [Ramamurthy et al., 2022] or replace RL.”
RewardBench: Evaluating Reward Models for Language Modeling (2024, methodology): “Given a reward model, a policy can be learned using RL algorithms like PPO (Schulman et al., 2017), a method that has been shown to work well for language policies (Ramamurthy et al., 2022).”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, unknown): “…(Brown et al., 2019; Shin et al., 2023) and language models (Ziegler et al., 2019; Stiennon et al., 2020; Wu et al., 2021; Nakano et al., 2021; Ouyang et al., 2022; Menick et al., 2022; Glaese et al., 2022; Gao et al., 2022; Bai et al., 2022a; Ganguli et al., 2022; Ramamurthy et al., 2022).”
Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning (2023, methodology): “Finetuning LLMs with RL Recent works successfully leveraged RL to finetune LLMs. RL was used in particular to improve alignment between generated text and human preferences [Stiennon et al., 2020, Ouyang et al., 2022, Ramamurthy et al., 2022].”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, methodology): “To address this issue, an open-source library named RL4LMs [49] has recently been introduced, consisting of building blocks for fine-tuning and evaluating RL algorithms on LM-based generation.”
Statistical Rejection Sampling Improves Preference Optimization (2023, background): “This approach has led to enhanced performance in various applications, including translation (Kreutzer et al., 2018), summarization (Stiennon et al., 2020), storytelling (Ziegler et al., 2019), and following instructions (Ramamurthy et al., 2022).”
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, background): “Previous works have explored the implementation details of PPO for LLMs (Zheng et al., 2023; Ramamurthy et al., 2023).”
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning (2022, background): “Unlike OpenAI variants, however, open source models have relatively fewer alignment procedures in place —though work in this area is emerging (Ramamurthy et al., 2022; Ganguli et al., 2022).”
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation (2023, unknown): “Our approach is not tied to the PPO algorithm; implementations may employ a variety of other RL algorithms, such as A2C [22] or NLPO [32].”
Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023, methodology): “Ramamurthy et al. (2022) demonstrate that RL generally outperforms SFT, but that their combination performs the best.”
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, methodology): “Unfortunately, previous studies have shown that SFT with a pre-determined dataset is usually of inferior performance Ramamurthy et al. (2022).”
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, methodology): “This is consistent with findings from previous RLHF studies ([29], [33]).”