Concrete Problems in AI Safety
By Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dandelion Mané. 2016
3,295 citations recorded by the source corpus.
Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems. We present a list of five practical research problems related to accident risk, categorized according to whether the problem originates from having the wrong objective function ("avoiding side effects" and "avoiding reward hacking"), an objective function that is too expensive to evaluate frequently ("scalable supervision"), or undesirable behavior during the learning process ("safe exploration" and "distributional shift"). We review previous work in these areas as well as suggesting research directions with a focus on relevance to cutting-edge AI systems. Finally, we consider the high-level question of how to think most productively about the safety of forward-looking applications of AI.
Subfields: Artificial Intelligence, Safety, Risk, Reliability and Quality, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Percy Liang
- Quoc V. Le
- Jianfeng Gao
- Pieter Abbeel
- Ruslan Salakhutdinov
In Safety, Risk, Reliability and Quality
- Yarin Gal
- Marcel A. J. van Gerven
- Gabrielle Ras
- James Sharp
- Xiaowei Huang
- Finale Doshi-Velez
- José Miguel Hernández-Lobato
- Bo Li
In Safety, Robustness & Interpretability
Citations in the corpus
- Shortcut learning in deep neural networks (2020, background): “However, they all too often contain unexpected shortcuts that allow for so-called reward hacking [68].”
- A General Language Assistant as a Laboratory for Alignment (2021, background): “There have been many works related to AI safety and alignment, including some suggestions for global research plans such as [AOS16] and [HCSS21].”
- Recursively Summarizing Books with Human Feedback (2021, background): “Successfully training ML systems on such tasks will require more scalable means of producing an effective training signal — this problem is known as scalable oversight (Amodei et al., 2016).”
- Alignment of Language Agents (2021, background): “The lack of alignment we consider can occur by accident (Amodei et al., 2016), resulting from the system designer making a mistake in their specification for the system.”
- Scalable agent alignment via reward modeling: a research direction (2018, background): “If the reward model only captures most aspects of the objective but not all of it, this can lead the agent to find undesirable degenerate solutions (Amodei et al., 2016; Lehman et al., 2018; Ibarz et al., 2018).”
- Reward learning from human preferences and demonstrations in Atari (2018, background): “This so-called reward hacking problem (Amodei et al., 2016; Everitt, 2018) is not unique to reward learning; hard-coded reward functions are also exploitable in this way (Lehman et al.”
- Supervising strong learners by amplifying weak experts (2018, background): “But aggressively optimizing that proxy can lead to pathological behavior (Lehman et al., 2018; Amodei and Clark, 2016; Amodei et al., 2016), an example of Goodhart’s Law.”
- Ethical Challenges in Data-Driven Dialogue Systems (2017, methodology): “To examine safety in the context of dialogue systems, we adopt the perspective of (Amodei et al. 2016) and focus on the problem of accidents , defined as unintended or harmful behaviour resulting from poor design.”
- Constrained Policy Optimization (2017, background): “In domains like this, safe exploration for RL agents is important (Moldovan & Abbeel, 2012; Amodei et al., 2016).”
- Measuring Progress on Scalable Oversight for Large Language Models (2022, background): “…to develop robust techniques for scalable over-sight : the ability to provide reliable supervision—in the form of labels, reward signals, or critiques—to models in a way that will remain effective past the point that models start to achieve broadly human-level performance (Amodei et al., 2016).”
- Self-critiquing models for assisting human evaluators (2022, background): “This leads to the problem of scalable oversight [AOS16]: How can we effectively provide feedback to models on tasks that are difficult for humans to evaluate?”
- Red Teaming Language Models with Language Models (2022, background): “It is also important to conduct broad surveys of possible harms (Amodei et al., 2016; Bommasani et al., 2021; Hendrycks et al., 2021; Weidinger et al., 2021, inter alia), to minimize the number of unexpected harms.”
- AI safety via debate (2018, background): “Subtle problems in alignment can lead to unexpected and potentially unsafe behavior [Amodei et al., 2016], and we expect this problem to get worse as systems become more capable.”
- Discovering Language Model Behaviors with Model-Written Evaluations (2022, background): “Such training may lead to models tailoring responses to exploit quirks in the human evaluators to look preferable, rather than actually improving the responses (“sycophancy”; Cotra, 2021a), a form of reward hacking (Amodei et al., 2016).”
- What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2022, background): “While we do not ask how specifically respondents think this could happen, potential reasons for such concerns are discussed by Bostrom (2014); Amodei et al. (2016) and Hubinger et al. (2019).”
- Adversarial Training for High-Stakes Reliability (2022, methodology): “Most empirical work on techniques for aligning powerful AI [12, 13, 14, 15, 16] has focused on achieving good average-case performance in domains where no single action is catastrophic, for example using human trajectory rankings [17, 18, 19] or imitation learning [20, 21].”
- True Few-Shot Learning with Language Models (2021, methodology): “We evaluate the generalization loss of the algorithm chosen by CV (likewise for MDL): where Reliability is important for many practical or commercial applications where worst-case performance is important, such as image recognition [35, 36], dialogue systems [37, 38], and robotics [39, 40].”