AGI Safety Literature Review

By Tom Everitt, Gary Lea, Marcus Hutter. 2018

132 citations recorded by the source corpus.

The development of Artificial General Intelligence (AGI) promises to be a major event. Along with its many potential benefits, it also raises serious safety concerns. The intention of this paper is to provide an easily accessible and up-to-date collection of references for the emerging field of AGI safety. A significant number of safety problems for AGI have been identified. We list these, and survey recent research on solving them. We also cover works on how best to think of AGI from the limited knowledge we have today, predictions for when AGI will first be created, and what will happen after its creation. Finally, we review the current public policy on AGI.

Subfields: Artificial Intelligence, Safety Research, Social Psychology, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Social Psychology

In Safety, Robustness & Interpretability

Citations in the corpus

Alignment of Language Agents (2021, background): “Everitt et al. (2018) give a review of AGI safety literature, with both problems and design ideas for safe AGI, but again don’t focus on language.”
Artificial Intelligence, Values, and Alignment (2020, background): “As the budding field of technical AI safety has made clear, the creation of general AI could give rise to a discrete set of problems that require special safety measures and solutions (Everitt et al., 2018).”
Scalable agent alignment via reward modeling: a research direction (2018, background): “See Everitt et al. (2018) for more in-depth survey and literature review.”
The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities (2018, background): “For example, beyond their importance to digital evolution, the many examples of “misspecified fitness functions” in this article connect to the broader field of artificial intelligence safety: Many researchers therein are concerned with the potential for perverse outcomes from optimizing reward func”
Taxonomy of Risks posed by Language Models (2022, background): “While many of these risks intersect with longer-term concerns, we refer to other work [26, 53, 99] for more focused discussion of risks in the context of AGI safety and superintelligence.”
Modeling AGI Safety Frameworks with Causal Influence Diagrams (2019, background): “Although the development of AGI has many potential benefits, there are also many safety concerns that have been raised in the literature [Bostrom, 2014; Everitt et al., 2018; Amodei et al., 2016].”
“Good Robot!”: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer (2019, background): “Safety is crucial for reinforcement learning in many realworld settings [32], [33], [34].”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, background): “The question of whether and why we should create general AI is a complicated one and is the focus of many articles [143, 41, 4, 18, 15].”
The alignment problem from a deep learning perspective (2022, background): “For a more comprehensive overview, see Ngo [2022a] and other broad surveys and courses that include work relevant to alignment of AGI [Hendrycks et al., 2021, Hendrycks, 2023b, Amodei et al., 2016, Everitt et al., 2018].”
Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings (2019, background): “For example, a worry in the AI safety literature (Everitt et al., 2018) is that an agent finds a way to tamper with the reward signal, giving itself high reward without completing its intended goals.”