AI safety via debate

By Geoffrey Irving, Paul Christiano, Dario Amodei. 2018

428 citations recorded by the source corpus.

To make AI systems broadly useful for challenging real-world tasks, we need them to learn complex human goals and preferences. One approach to specifying complex goals asks humans to judge during training which agent behaviors are safe and useful, but this approach can fail if the task is too complicated for a human to directly judge. To help address this concern, we propose training agents via self play on a zero sum debate game. Given a question or proposed action, two agents take turns making short statements up to a limit, then a human judges which of the agents gave the most true, useful information. In an analogy to complexity theory, debate with optimal play can answer any question in PSPACE given polynomial time judges (direct judging answers only NP questions). In practice, whether debate works involves empirical questions about humans and the tasks we want AIs to perform, plus theoretical questions about the meaning of AI alignment. We report results on an initial MNIST experiment where agents compete to convince a sparse classifier, boosting the classifier's accuracy from 59.4% to 88.9% given 6 pixels and from 48.2% to 85.2% given 4 pixels. Finally, we discuss theoretical and practical aspects of the debate model, focusing on potential weaknesses as the model scales up, and we propose future human and computer experiments to test these properties.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Safety, Robustness & Interpretability

Citations in the corpus

Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “Scaling supervision has been widely discussed as a possibility for AI alignment, with specific proposals such as [Christiano et al., 2018, Irving et al., 2018] and recent empirical work like [Bowman et al., 2022].”
Fine-Tuning Language Models from Human Preferences (2019, background): “…interactive communication between humans and ML models is a requirement for scalable reward learning methods such as ampliﬁcation, debate, and recursive reward modeling (Christiano et al., 2018; Irving et al., 2018; Leike et al., 2018), and natural language is how humans communicate complex ideas.”
WebGPT: Browser-assisted question-answering with human feedback (2021, methodology): “We could mitigate this using methods like debate [Irving et al., 2018], in which models are trained to find evidence both for and against different claims.”
Finding Generalizable Evidence by Learning to Convince Q&A Models (2019, background): “Debate Evidence extraction can be viewed as a form of debate, in which multiple agents support different stances (Irving et al., 2018; Irving and Askell, 2019).”
Recursively Summarizing Books with Human Feedback (2021, background): “Christiano et al. (2018), Irving et al. (2018), and Leike et al. (2018) all propose in some way reducing the task into simpler parts which humans can supervise.”
Alignment of Language Agents (2021, background): “One approach to mitigate deception is debate (Irving et al., 2018) which sets up a game in which a debate between two agents is presented to a human judge, who awards the winner.”
Artificial Intelligence, Values, and Alignment (2020, background): “…where the agent discovers ingenious ways to achieve its objective or reward even though they differ from what was intended, and how to evaluate the performance of agents whose cognitive abilities potentially signiﬁcantly exceed our own (Irving et al., 2018; Leike et al., 2018; Christiano, 2016).”
Scalable agent alignment via reward modeling: a research direction (2018, background): “This analogy to complexity theory, ﬁrst introduced by Irving et al. (2018), provides two important insights: 1.”
Supervising strong learners by amplifying weak experts (2018, background): “Debate: training AI systems to debate each other (Irving et al., 2018) is another possible approach to training question-answering systems where a human expert cannot evaluate answers directly.”
Measuring Progress on Scalable Oversight for Large Language Models (2022, background): “Proposed scalable oversight techniques like debate (Irving et al., 2018) or market making (Hubinger, 2020) offer more sophisticated options for attacking this problem and could provide leverage on the problem, but none have been proven out empirically.”
Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “Prior work has suggested that assistance from ML systems may help with accurate human supervision (Christiano et al., 2018; Irving et al., 2018; Leike et al., 2018).”
Self-critiquing models for assisting human evaluators (2022, background): “[ICA18] introduce debate and implement a toy version with sparse MNIST classification.”
Adversarial Training for High-Stakes Reliability (2022, background): “Adversarial training plays an important role in many proposals for aligning powerful AI systems [25, 26, 27, 28].”
Teaching language models to support answers with verified quotes (2022, background): “…of a source is itself a challenging task, augmenting the setup with a way to help the human make good judgements using techniques like amplification (Christiano et al., 2018), recursive reward modelling (Leike et al., 2018), or debate (Irving et al., 2018) may offer a promising way forward.”
GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023, neutral): “Our non-expert validation phase is designed to ensure that any experiments evaluating scalable oversight methods like debate (Irving et al., 2018), market-making (Hubinger, 2020), recursive reward modeling (Leike et al., 2018), etc., will not have trivial solutions as the non-experts are given more…”
Unsolved Problems in ML Safety (2021, background): “Researchers could determine the veracity of model assertions, possibly through an adversarial truth-ﬁnding process [93].”