Reward learning from human preferences and demonstrations in Atari
By Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, Dario Amodei. 2018
490 citations recorded by the source corpus.
To solve complex real-world problems with reinforcement learning, we cannot rely on manually specified reward functions. Instead, we can have humans communicate an objective to the agent directly. In this work, we combine two approaches to learning from human feedback: expert demonstrations and trajectory preferences. We train a deep neural network to model the reward function and use its predicted reward to train an DQN-based deep reinforcement learning agent on 9 Atari games. Our approach beats the imitation learning baseline in 7 games and achieves strictly superhuman performance on 2 games without using game rewards. Additionally, we investigate the goodness of fit of the reward model, present some reward hacking problems, and study the effects of noise in the human labels.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Alignment & RLHF, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Mohit Bansal
- Quoc V. Le
- Ruslan Salakhutdinov
- Percy Liang
- Chelsea Finn
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Jitendra Malik
- Xiaogang Wang
- Ming-Hsuan Yang
- Andrew Zisserman
- Andrea Vedaldi
In Alignment & RLHF
- Geoffrey Irving
- Jan Leike
- Angelica Chen
- Sergey Levine
- Iason Gabriel
- Tomasz Korbak
- Anca Dragan
- Tom Everitt
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Yuxi Li
- Shimon Whiteson
- Shixiang Shane Gu
- Timothy P. Lillicrap
Citations in the corpus
- Training language models to follow instructions with human feedback (2022, methodology): “Originally developed for training simple robots in simulated environments and Atari games (Christiano et al., 2017; Ibarz et al., 2018), it has recently been applied to fine-tuning language models to summarize text (Ziegler et al., 2019; Stiennon et al., 2020; Böhm et al., 2019; Wu et al., 2021).”
- Fine-Tuning Language Models from Human Preferences (2019, background): “…is a long history of work learning such models from humans through interaction, this work has only recently been applied to modern deep learning, and even then has only been applied to relatively simple simulated environments (Christiano et al., 2017; Ibarz et al., 2018; Bahdanau et al., 2018).”
- A General Language Assistant as a Laboratory for Alignment (2021, neutral): “• In marked contrast to the scaling results mentioned earlier, where PM scales best on hierarchically ranked datasets, we find that it’s better for the PMP stage of training to focus on binary discrimination (see figure 18).”
- Learning to summarize from human feedback (2020, background): “In a related line of research, human feedback has been used to train agents in simulated environments [10, 24].”
- Recursively Summarizing Books with Human Feedback (2021, methodology): “…et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler, 2018), story generation (Zhou and Xu, 2020), review generation (Cho et al., 2018), and evidence extraction (Perez et al., 2019), and agents in simulated environments (Christiano et al., 2017; Ibarz et al., 2018).”
- Alignment of Language Agents (2021, background): “Having online feedback to iteratively update the reward model throughout agent training can correct for this somewhat (Ibarz et al., 2018), but its application is hard to do practically, as it requires carefully balancing the frequency of updates of the learnt objective and the optimizing system.”
- Scalable agent alignment via reward modeling: a research direction (2018, methodology): “Reinforcement learning from a learned reward function has been successful in gridworlds (Bahdanau et al., 2018), Atari games (Christiano et al., 2017; Ibarz et al., 2018), and continuous motor control tasks (Ho & Ermon, 2016; Christiano et al., 2017).”
- Scaling Laws for Reward Model Overoptimization (2022, methodology): “Reinforcement learning from human feedback [Christiano et al., 2017, Ibarz et al., 2018] has been used broadly in language models [Stiennon et al., 2020, Ouyang et al., 2022, Nakano et al., 2021, Bai et al., 2022].”
- Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, background): “…PbRL was introduced almost a decade ago, most work in it has been primarily ap- plied or experimental in nature (Riad et al., 2012; Jain et al., 2013; Busa-Fekete et al., 2014; Christiano et al., 2017; Wirth and Fürnkranz, 2013; Wirth et al., 2016, 2017; Ibarz et al., 2018; Kupcsik et al., 2018).”
- Risks from Learned Optimization in Advanced Machine Learning Systems (2019, background): “An interesting approach to analyzing this connection is presented in Ibarz et al, where empirical samples of the true reward and a learned reward on the same trajectories are used to create a scatter-plot visualization of the alignment between the two.[18] The assumption in that work is that a monot”
- The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “Reward hacking , or the gaming of misspecified reward functions by RL agents, has appeared in a variety of contexts, such as game playing (Ibarz et al., 2018), text summarization (Paulus et al., 2018), and autonomous driving (Knox et al., 2021).”
- A Framework for Data-Driven Robotics (2019, background): “Preferences can be extracted with whole episode comparisons [54]–[57] or shorter clip comparisons [58], [59].”
- Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, methodology): “We used an architecture for reward learning similar to the one proposed in (Ibarz et al., 2018), with four convolutional layers with sizes 7x7, 5x5, 3x3, and 3x3, with strides 3, 2, 1, and 1.”
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “…et al., 2022b, Saunders et al., 2022, Ye et al., 2023, Kim et al., 2023, Akyürek et al., 2023, Madaan et al., 2023, Chen et al., 2023, Gilardi et al., 2023] with research agendas like Recursive Reward Modeling [Leike et al., 2018] and AI Safety via debate [Irving et al., 2018, Du et al., 2023].”
- Scaling data-driven robotics with reward sketching and batch reinforcement learning (2019, background): “Preferences can be elicited with whole episode comparisons [2, 3, 14, 18] or shorter clip comparisons [15, 29].”
- Reward Design with Language Models (2023, background): “Reward hacking has appeared in various domains such as autonomous driving (Knox et al., 2021) and game-playing (Ibarz et al., 2018).”
- A data-driven approach for learning to control computers (2022, background): “Interest in human alignment has accelerated (Wirth et al., 2017; Ibarz et al., 2018), and spread to a variety of domains, including dialogue agents (Jaques et al.”
- Defining and Characterizing Reward Hacking (2022, unknown): “…a robot positioning itself between the camera and the object it is supposed to grasp in a way that tricks the reward model (Amodei et al., 2017), the previously mentioned boat race example (Clark and Amodei, 2016), and a multitude of examples of reward model hacking in Atari (Ibarz et al., 2018).”