Supervising strong learners by amplifying weak experts

By Paul Christiano, Buck Shlegeris, Dario Amodei. 2018

178 citations recorded by the source corpus.

Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an alternative training strategy which progressively builds up a training signal for difficult problems by combining solutions to easier subproblems. Iterated Amplification is closely related to Expert Iteration (Anthony et al., 2017; Silver et al., 2017), except that it uses no external reward function. We present results in algorithmic environments, showing that Iterated Amplification can efficiently learn complex behaviors.

Subfields: Artificial Intelligence, Management Science and Operations Research, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Percy Liang
  4. Jianfeng Gao
  5. Sergey Levine
  6. Yoshua Bengio
  7. Quoc V. Le
  8. Kyunghyun Cho

In Management Science and Operations Research

  1. Sergey Levine
  2. Pieter Abbeel
  3. Anca Dragan
  4. John Schulman
  5. Shane Legg
  6. Tom Everitt
  7. Jan Leike
  8. Chelsea Finn

In Alignment & RLHF

  1. Geoffrey Irving
  2. Angelica Chen
  3. Ethan Perez
  4. Jan Leike
  5. Sergey Levine
  6. Richard Ngo
  7. Iason Gabriel
  8. Anca Dragan

Citations in the corpus