Deep Reinforcement Learning from Human Preferences
By Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei. Neural Information Processing Systems, 2017
5,646 citations recorded by the source corpus.
For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.
Subfields: Artificial Intelligence, Control and Systems Engineering, Alignment & RLHF, Reinforcement Learning
View Paper: https://arxiv.org/abs/1706.03741
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohit Bansal
- Yoshua Bengio
- Percy Liang
- Jianfeng Gao
- Luke S. Zettlemoyer
- Quoc V. Le
- Ruslan Salakhutdinov
In Control and Systems Engineering
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Abhishek Gupta
- Peter Stone
- Vikash Kumar
- Scott Niekum
- Silvio Savarese
In Alignment & RLHF
- Geoffrey Irving
- Angelica Chen
- Jan Leike
- Ethan Perez
- Richard Ngo
- Sergey Levine
- Danqi Chen
- Tom Everitt
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shixiang Shane Gu
- Shimon Whiteson
- Yuxi Li
- Abhishek Gupta
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “RLHF has emerged as a powerful strategy for fine-tuning Large Language Models, enabling significant improvements in their performance (Christiano et al., 2017).”
- Training language models to follow instructions with human feedback (2022, methodology): “Specifically, we use reinforcement learning from human feedback (RLHF; Christiano et al., 2017; Stiennon et al., 2020) to fine-tune GPT-3 to follow a broad class of written instructions (see Figure 2).”
- Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “The method therefore improves upon, and partially replaces reinforcement learning from human feedback [Christiano et al., 2017].”
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “While the most straightforward approach to preference learning is supervised fine-tuning on human demonstrations of high quality responses, the most successful class of methods is reinforcement learning from human (or AI) feedback (RLHF/RLAIF; [12, 2]).”
- Fine-Tuning Language Models from Human Preferences (2019, background): “Following Christiano et al. (2017), we ask human labelers to pick which of several values of yi is the best response to a given input x.”
- OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “Finally, there are also differences in the fine-tuning algorithms used, for example, some of the OpenAI davinci models use RLHF (Christiano et al., 2017) on feedback signals gathered from their API in addition to supervised fine-tuning.”
- The False Promise of Imitating Proprietary LLMs (2023, background): “However, it also may be possible to use the target model to perform RLHF or constitutional AI (OpenAI, 2022; Christiano et al., 2017; Bai et al., 2022) to further improve results.”
- The Capacity for Moral Self-Correction in Large Language Models (2023, methodology): “We study decoder-only transformer models fine-tuned with Reinforcement Learning from Human Feedback (RLHF) [13, 57] to function as helpful dialogue models.”
- Augmented Language Models: a Survey (2023, background): “Besides learning to use external tools, RLHF has also proven useful for a wide range of language generation tasks, from summarization (Ziegler et al., 2019; Wu et al., 2021; Stiennon et al., 2020) to training more helpful, harmless, and accurate assistants (Glaese et al., 2022; Cohen et al., 2022; O”
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, methodology): “We do not define what “harmful” means, as this is a complex and subjective concept; instead, we rely on the red team to make their own determinations via a pairwise preference choice [14].”
- A General Language Assistant as a Laboratory for Alignment (2021, background): “Such models can be used as filters, and they can also be leveraged more powerfully as preference models for reinforcement learning from human feedback (RLHF) [CLB + 17], in order to train aligned policies.”
- Learning to summarize from human feedback (2020, background): “In a related line of research, human feedback has been used to train agents in simulated environments [10, 24].”
- Recursively Summarizing Books with Human Feedback (2021, methodology): “…et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler, 2018), story generation (Zhou and Xu, 2020), review generation (Cho et al., 2018), and evidence extraction (Perez et al., 2019), and agents in simulated environments (Christiano et al., 2017; Ibarz et al., 2018).”
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, background): “[7]) or submitting a feedback report, does not scale.”
- Scalable agent alignment via reward modeling: a research direction (2018, methodology): “Reinforcement learning from a learned reward function has been successful in gridworlds (Bahdanau et al., 2018), Atari games (Christiano et al., 2017; Ibarz et al., 2018), and continuous motor control tasks (Ho & Ermon, 2016; Christiano et al., 2017).”
- Reward learning from human preferences and demonstrations in Atari (2018, methodology): “The direct predecessor of our work is Christiano et al. (2017), with similar tasks, rewards, policy architectures, and preference learning scheme.”
- Supervising strong learners by amplifying weak experts (2018, background): “…box (RL from a human training signal) is beginning to be explored, and includes inverse reinforcement learning (Ng and Russell, 2000; Abbeel and Ng, 2004; Finn et al., 2016) and RL from human feedback (Knox and Stone, 2009; Pilarski et al., 2011; MacGlashan et al., 2017; Christiano et al., 2017).”
- Learning to Understand Goal Specifications by Modelling Reward (2018, methodology): “Besides IRL-style approaches, other apprenticeship learning methods involve training a policy (Knox & Stone, 2009; Warnell et al., 2017) or a reward function (Wilson et al., 2012; Christiano et al., 2017) directly from human feedback.”