Scalable agent alignment via reward modeling: a research direction
By Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, Shane Legg. arXiv.org, 2018
615 citations recorded by the source corpus.
One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions? We outline a high-level research direction to solve the agent alignment problem centered around reward modeling: learning a reward function from interaction with the user and optimizing the learned reward function with reinforcement learning. We discuss the key challenges we expect to face when scaling reward modeling to complex and general domains, concrete approaches to mitigate these challenges, and ways to establish trust in the resulting agents.
Subfields: Artificial Intelligence, Alignment & RLHF, Reinforcement Learning
View Paper: https://arxiv.org/abs/1811.07871
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Sergey Levine
- Percy Liang
- Jianfeng Gao
- Luke S. Zettlemoyer
- Quoc V. Le
- Yoshua Bengio
- Kyunghyun Cho
In Alignment & RLHF
- Angelica Chen
- Geoffrey Irving
- Ethan Perez
- Jan Leike
- Richard Ngo
- Danqi Chen
- Sergey Levine
- Tomasz Korbak
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shimon Whiteson
- Shixiang Shane Gu
- Mohammad Norouzi
- Yuxi Li
Citations in the corpus
- Training language models to follow instructions with human feedback (2022, positive): “We make progress on aligning language models by training them to act in accordance with the user’s intention (Leike et al., 2018).”
- Fine-Tuning Language Models from Human Preferences (2019, background): “…interactive communication between humans and ML models is a requirement for scalable reward learning methods such as amplification, debate, and recursive reward modeling (Christiano et al., 2018; Irving et al., 2018; Leike et al., 2018), and natural language is how humans communicate complex ideas.”
- WebGPT: Browser-assisted question-answering with human feedback (2021, background): “Such setups can also be viewed as simple cases of recursive reward modeling [Leike et al., 2018] and Iterated Amplification [Christiano et al., 2018], in which the model assists its own evaluation.”
- Learning to summarize from human feedback (2020, background): “We follow the works of [3, 73], who fine-tune language models from human feedback using reward learning [35].”
- Recursively Summarizing Books with Human Feedback (2021, methodology): “We expect this to be a critical part of the alignment problem because we need to make sure humans can communicate their values to AI systems as they take on more societally-relevant tasks (Leike et al., 2018).”
- Alignment of Language Agents (2021, methodology): “Recursive Reward Modeling (Leike et al., 2018) proposes to use a sequence of agents trained using”
- Artificial Intelligence, Values, and Alignment (2020, background): “More recently, the Asilomar AI Principles hold that, ‘Highly autonomous AI systems should be designed so that their goals and behaviours can be assured to align with human values throughout their operation’ (Asilomar A.I., 2018).”
- Measuring Progress on Scalable Oversight for Large Language Models (2022, background): “• Self-Critique, Amplification and Recursive Reward Modeling: In this family of approaches (Leike et al., 2018; Christiano et al., 2018; Wu et al., 2021; Saunders et al., 2022), loosely speaking, the human participants iteratively (i) supervise a model using a human-supervision-based technique like…”
- Scaling Laws for Reward Model Overoptimization (2022, methodology): “It is also a first step towards recursive reward modelling [Leike et al., 2018], an approach towards reducing the additional source of overoptimization described in section 4.5, though it is subject to some theoretical limitations [Christiano et al., 2021].”
- Improving alignment of dialogue agents via targeted human judgements (2022, background): “Prior work has suggested that assistance from ML systems may help with accurate human supervision (Christiano et al., 2018; Irving et al., 2018; Leike et al., 2018).”
- Self-critiquing models for assisting human evaluators (2022, methodology): “Overall, we believe there is potential to do empirical experiments for scalable oversight with today’s models, using schemes similar to reward modeling [LKE18] or debate [IA19].”
- Adversarial Training for High-Stakes Reliability (2022, background): “Adversarial training plays an important role in many proposals for aligning powerful AI systems [25, 26, 27, 28].”
- Risks from Learned Optimization in Advanced Machine Learning Systems (2019, background): “4Leike et al.[8] introduce the concept of an objective recovered from perfect IRL.”
- Teaching language models to support answers with verified quotes (2022, background): “, 2018), recursive reward modelling (Leike et al., 2018), or debate (Irving et al.”
- A Hazard Analysis Framework for Code Synthesis Large Language Models (2022, neutral): “…as the degree to which the behavior of the AI does or does not accord with user intentions; misaligned AI may produce unsafe behavior) [3, 13, 24] • System Design and Implementation (e.g., UX/UI, Documentation, Requirements, Data Provenance, Validation) • Regulatory and Legal Oversight…”
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023, neutral): “…any experiments evaluating scalable oversight methods like debate (Irving et al., 2018), market-making (Hubinger, 2020), recursive reward modeling (Leike et al., 2018), etc., will not have trivial solutions as the non-experts are given more resources (which make the experiments more realistic);…”
- Unsolved Problems in ML Safety (2021, background): “, 2018 [120] considers safety research directions in reward modeling.”
- CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation (2024, positive): “Instruction Tuning: Instruction tuning (Wei et al., 2022) is a type of full fine-tuning that optimizes a model to produce more relevant answers to questions or instructions (Leike et al., 2018; Askell et al., 2021).”
- Modeling AGI Safety Frameworks with Causal Influence Diagrams (2019, background): “For example, both reward modeling [Leike et al., 2018] and cooperative inverse RL [HadfieldMenell et al.”