Supervising strong learners by amplifying weak experts
By Paul Christiano, Buck Shlegeris, Dario Amodei. 2018
178 citations recorded by the source corpus.
Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an alternative training strategy which progressively builds up a training signal for difficult problems by combining solutions to easier subproblems. Iterated Amplification is closely related to Expert Iteration (Anthony et al., 2017; Silver et al., 2017), except that it uses no external reward function. We present results in algorithmic environments, showing that Iterated Amplification can efficiently learn complex behaviors.
Subfields: Artificial Intelligence, Management Science and Operations Research, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Sergey Levine
- Yoshua Bengio
- Quoc V. Le
- Kyunghyun Cho
In Management Science and Operations Research
In Alignment & RLHF
- Geoffrey Irving
- Angelica Chen
- Ethan Perez
- Jan Leike
- Sergey Levine
- Richard Ngo
- Iason Gabriel
- Anca Dragan
Citations in the corpus
- Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “Scaling supervision has been widely discussed as a possibility for AI alignment, with specific proposals such as [Christiano et al., 2018, Irving et al., 2018] and recent empirical work like [Bowman et al., 2022].”
- Fine-Tuning Language Models from Human Preferences (2019, background): “…interactive communication between humans and ML models is a requirement for scalable reward learning methods such as amplification, debate, and recursive reward modeling (Christiano et al., 2018; Irving et al., 2018; Leike et al., 2018), and natural language is how humans communicate complex ideas.”
- WebGPT: Browser-assisted question-answering with human feedback (2021, background): “Such setups can also be viewed as simple cases of recursive reward modeling [Leike et al., 2018] and Iterated Amplification [Christiano et al., 2018], in which the model assists its own evaluation.”
- A General Language Assistant as a Laboratory for Alignment (2021, background): “Furthermore, some proposals [CSA18, ICA18] for aligning more advanced AIs use different models to train or evaluate each other, so that the effectiveness and reliability of these techniques may ultimately depend on the performance and robustness of preference models.”
- Learning to summarize from human feedback (2020, background): “One approach is to train ML systems to help humans perform the evaluation task quickly and accurately [9].”
- Recursively Summarizing Books with Human Feedback (2021, methodology): “While Dayan and Hinton (1993) and Christiano et al. (2018) only tried this on simple algorithmic tasks, Perez et al. (2020) and Rajani et al. (2019) use similar ideas for question-answering tasks using a single step of decomposition.”
- Alignment of Language Agents (2021, background): “Iterated Amplification (Christiano et al., 2018) progressively builds up a training signal for hard problems by decomposing the problem into subproblems, then combining solutions to easier subproblems.”
- Scalable agent alignment via reward modeling: a research direction (2018, background): “This expert reasoning can then be improved and accelerated (Christiano et al., 2018; Evans et al., 2018; Stuhlmüller, 2018).”
- Measuring Progress on Scalable Oversight for Large Language Models (2022, background): “• Self-Critique, Amplification and Recursive Reward Modeling: In this family of approaches (Leike et al., 2018; Christiano et al., 2018; Wu et al., 2021; Saunders et al., 2022), loosely speaking, the human participants iteratively (i) supervise a model using a human-supervision-based technique like…”
- Improving alignment of dialogue agents via targeted human judgements (2022, background): “Prior work has suggested that assistance from ML systems may help with accurate human supervision (Christiano et al., 2018; Irving et al., 2018; Leike et al., 2018).”
- Self-critiquing models for assisting human evaluators (2022, background): “[CSA18] implement iterative amplification for algorithmic tasks.”
- AI safety via debate (2018, background): “Similarly, amplification [Christiano et al., 2018] trains a module to generate subquestions as part of training an answerer, but only the answerer is needed at test time (see Section 7).”
- TALM: Tool Augmented Language Models (2022, background): “Iterative self-play is related to expert iteration [Anthony et al., 2017], which has been demonstrated to work well in tasks with extremely weak supervision [Christiano et al., 2018].”
- Unsupervised Question Decomposition for Question Answering (2020, background): “This approach leverages strong performance on simple questions to help answer harder questions (Christiano et al., 2018).”
- Risks from Learned Optimization in Advanced Machine Learning Systems (2019, methodology): “18Iterated Distillation and Amplification might be a possible solution to problems of this form.”
- Teaching language models to support answers with verified quotes (2022, methodology): “…of a source is itself a challenging task, augmenting the setup with a way to help the human make good judgements using techniques like amplification (Christiano et al., 2018), recursive reward modelling (Leike et al., 2018), or debate (Irving et al., 2018) may offer a promising way forward.”