Deep Reinforcement Learning from Human Preferences

By Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei. Neural Information Processing Systems, 2017

5,646 citations recorded by the source corpus.

For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.

Subfields: Artificial Intelligence, Control and Systems Engineering, Alignment & RLHF, Reinforcement Learning

View Paper: https://arxiv.org/abs/1706.03741

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Mohit Bansal
  3. Yoshua Bengio
  4. Percy Liang
  5. Jianfeng Gao
  6. Luke S. Zettlemoyer
  7. Quoc V. Le
  8. Ruslan Salakhutdinov

In Control and Systems Engineering

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Abhishek Gupta
  5. Peter Stone
  6. Vikash Kumar
  7. Scott Niekum
  8. Silvio Savarese

In Alignment & RLHF

  1. Geoffrey Irving
  2. Angelica Chen
  3. Jan Leike
  4. Ethan Perez
  5. Richard Ngo
  6. Sergey Levine
  7. Danqi Chen
  8. Tom Everitt

In Reinforcement Learning

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Ofir Nachum
  5. Shixiang Shane Gu
  6. Shimon Whiteson
  7. Yuxi Li
  8. Abhishek Gupta

Citations in the corpus