Reward learning from human preferences and demonstrations in Atari

By Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, Dario Amodei. 2018

490 citations recorded by the source corpus.

To solve complex real-world problems with reinforcement learning, we cannot rely on manually specified reward functions. Instead, we can have humans communicate an objective to the agent directly. In this work, we combine two approaches to learning from human feedback: expert demonstrations and trajectory preferences. We train a deep neural network to model the reward function and use its predicted reward to train an DQN-based deep reinforcement learning agent on 9 Atari games. Our approach beats the imitation learning baseline in 7 games and achieves strictly superhuman performance on 2 games without using game rewards. Additionally, we investigate the goodness of fit of the reward model, present some reward hacking problems, and study the effects of noise in the human labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Alignment & RLHF, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Yoshua Bengio
  3. Pieter Abbeel
  4. Mohit Bansal
  5. Quoc V. Le
  6. Ruslan Salakhutdinov
  7. Percy Liang
  8. Chelsea Finn

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Trevor Darrell
  3. Dacheng Tao
  4. Jitendra Malik
  5. Xiaogang Wang
  6. Ming-Hsuan Yang
  7. Andrew Zisserman
  8. Andrea Vedaldi

In Alignment & RLHF

  1. Geoffrey Irving
  2. Jan Leike
  3. Angelica Chen
  4. Sergey Levine
  5. Iason Gabriel
  6. Tomasz Korbak
  7. Anca Dragan
  8. Tom Everitt

In Reinforcement Learning

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Ofir Nachum
  5. Yuxi Li
  6. Shimon Whiteson
  7. Shixiang Shane Gu
  8. Timothy P. Lillicrap

Citations in the corpus