Is Power-Seeking AI an Existential Risk?

By J. Carlsmith. 2022

163 citations recorded by the source corpus.

This report examines what I see as the core argument for concern about existential risk from misaligned artificial intelligence. I proceed in two stages. First, I lay out a backdrop picture that informs such concern. On this picture, intelligent agency is an extremely powerful force, and creating agents much more intelligent than us is playing with fire -- especially given that if their objectives are problematic, such agents would plausibly have instrumental incentives to seek power over humans. Second, I formulate and evaluate a more specific six-premise argument that creating agents of this kind will lead to existential catastrophe by 2070. On this argument, by 2070: (1) it will become possible and financially feasible to build relevantly powerful and agentic AI systems; (2) there will be strong incentives to do so; (3) it will be much harder to build aligned (and relevantly powerful/agentic) AI systems than to build misaligned (and relevantly powerful/agentic) AI systems that are still superficially attractive to deploy; (4) some such misaligned systems will seek power over humans in high-impact ways; (5) this problem will scale to the full disempowerment of humanity; and (6) such disempowerment will constitute an existential catastrophe. I assign rough subjective credences to the premises in this argument, and I end up with an overall estimate of ~5% that an existential catastrophe of this kind will occur by 2070. (May 2022 update: since making this report public in April 2021, my estimate here has gone up, and is now at>10%.)

Subfields: Economics and Econometrics, Safety Research, Sociology and Political Science, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Safety Research

In Sociology and Political Science

In Safety, Robustness & Interpretability

Citations in the corpus

Scaling Laws for Reward Model Overoptimization (2022, background): “, 2015), and to result in existentially catastrophic outcomes (Carlsmith, 2022; Cotra, 2022).”
Large Language Model Alignment: A Survey (2023, background): “This trajectory could culminate in the full disempowerment of humanity, inevitably leading to catastrophic outcomes for human existence (Carlsmith, 2022).”
The alignment problem from a deep learning perspective (2022, background): “Previous writings have argued that AGIs will be highly challenging to robustly align, and that misaligned AGIs may pose risks on a sufﬁciently large scale to threaten human civilization [Russell, 2019, Bostrom, 2014, Yudkowsky, 2016, Carlsmith, 2022, Cohen et al., 2022].”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “While none of the features we probed for were especially critical, the Pile subset context neurons could be considered a precursor for situational awareness [47, 82].”