An Overview of Catastrophic AI Risks

By Dan Hendrycks, Mantas Mazeika, Thomas Woodside. 2023

315 citations recorded by the source corpus.

Rapid advancements in artificial intelligence (AI) have sparked growing concerns among experts, policymakers, and world leaders regarding the potential for increasingly advanced AI systems to pose catastrophic risks. Although numerous risks have been detailed separately, there is a pressing need for a systematic discussion and illustration of the potential dangers to better inform efforts to mitigate them. This paper provides an overview of the main sources of catastrophic AI risks, which we organize into four categories: malicious use, in which individuals or groups intentionally use AIs to cause harm; AI race, in which competitive environments compel actors to deploy unsafe AIs or cede control to AIs; organizational risks, highlighting how human factors and complex systems can increase the chances of catastrophic accidents; and rogue AIs, describing the inherent difficulty in controlling agents far more intelligent than humans. For each category of risk, we describe specific hazards, present illustrative stories, envision ideal scenarios, and propose practical suggestions for mitigating these dangers. Our goal is to foster a comprehensive understanding of these risks and inspire collective and proactive efforts to ensure that AIs are developed and deployed in a safe manner. Ultimately, we hope this will allow us to realize the benefits of this powerful technology while minimizing the potential for catastrophic outcomes.

Subfields: Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety Research

1. Richard Ngo

In Safety, Robustness & Interpretability

1. Zhexin Zhang
2. Hao Sun
3. Minlie Huang
4. Richard Ngo
5. Xinwei Wu
6. Wes Gurnee
7. Tianhao Shen
8. Jiawen Deng

Citations in the corpus

• Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “They might interact un-predictably with humans and other systems, especially in open environments (Hendrycks et al., 2023).”
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, unknown): “However, there has been growing concern from researchers, regulators, and industry leaders over the risk of malicious use posed by current and future AI systems (Brundage et al., 2018; Hendrycks et al., 2023; Executive Office of the President, 2023).”
• Rethinking machine unlearning for large language models (2024, unknown): “…of the existing literature on unlearning (Bourtoule et al., 2021; Jia et al., 2023; Kurmanji et al., 2023), and its progression in LLMs (Pawelczyk et al., 2023; Yao et al., 2023; Ishibashi & Shimodaira, 2023; Maini et al., 2024; Li et al., 2024a), we define the problem of LLM unlearning below.”
• Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023, background): “…caused concern that AI systems that we cannot trust are being widely deployed in the economy and in our lives, introducing a number of novel risks (Hendrycks et al., 2023), including potential future risks that AIs might deceive humans in order to accomplish undesirable goals (Ngo et al., 2022).”
• The alignment problem from a deep learning perspective (2022, background): “An increasing body of research aims to proactively address the alignment problem, motivated in large part by the desire to avoid hypothesized large-scale risks from AGIs that pursue unintended goals [OpenAI, 2023c, Hendrycks et al., 2023, Amodei et al., 2016, Hendrycks et al., 2021].”
• WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, background): “Systems using modern Language Models (LMs) pose considerable risks if not adequately safe-guarded [4, 8, 42, 15, 37].”
• Language Models Represent Space and Time (2023, background): “Better understanding of if and how LLMs model the world is critical for reasoning about the robustness, fairness, and safety of current and future AI systems (Bender et al., 2021; Weidinger et al., 2022; Bommasani et al., 2021; Hendrycks et al., 2023; Ngo et al., 2023).”
• Large Language Model Alignment: A Survey (2023, background): “, 2021) while the latter future potential risks associated with advanced LLMs (Hendrycks et al., 2023).”
• Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, background): “…to outweigh the externalities of potential negative outcomes (Hendrycks and Mazeika, 2022), as for example transparency plays an important role in both specific (Hubinger, 2020) and portfolio-based (Hendrycks, Mazeika, and Woodside, 2023) approaches to ensuring the safe development of AI systems.”