Scalable agent alignment via reward modeling: a research direction

By Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, Shane Legg. arXiv.org, 2018

615 citations recorded by the source corpus.

One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions? We outline a high-level research direction to solve the agent alignment problem centered around reward modeling: learning a reward function from interaction with the user and optimizing the learned reward function with reinforcement learning. We discuss the key challenges we expect to face when scaling reward modeling to complex and general domains, concrete approaches to mitigate these challenges, and ways to establish trust in the resulting agents.

Subfields: Artificial Intelligence, Alignment & RLHF, Reinforcement Learning

View Paper: https://arxiv.org/abs/1811.07871

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Sergey Levine
  3. Percy Liang
  4. Jianfeng Gao
  5. Luke S. Zettlemoyer
  6. Quoc V. Le
  7. Yoshua Bengio
  8. Kyunghyun Cho

In Alignment & RLHF

  1. Angelica Chen
  2. Geoffrey Irving
  3. Ethan Perez
  4. Jan Leike
  5. Richard Ngo
  6. Danqi Chen
  7. Sergey Levine
  8. Tomasz Korbak

In Reinforcement Learning

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Ofir Nachum
  5. Shimon Whiteson
  6. Shixiang Shane Gu
  7. Mohammad Norouzi
  8. Yuxi Li

Citations in the corpus