The alignment problem from a deep learning perspective
By Richard Ngo. 2022
333 citations recorded by the source corpus.
In coming years or decades, artificial general intelligence (AGI) may surpass human capabilities across many critical domains. We argue that, without substantial effort to prevent it, AGIs could learn to pursue goals that are in conflict (i.e. misaligned) with human interests. If trained like today's most capable models, AGIs could learn to act deceptively to receive higher reward, learn misaligned internally-represented goals which generalize beyond their fine-tuning distributions, and pursue those goals using power-seeking strategies. We review emerging evidence for these properties. In this revised paper, we include more direct empirical evidence published as of early 2025. AGIs with these properties would be difficult to align and may appear aligned even when they are not. Finally, we briefly outline how the deployment of misaligned AGIs might irreversibly undermine human control over the world, and we review research directions aimed at preventing this outcome.
Subfields: Computer Vision and Pattern Recognition, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Bernt Schiele
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Jitendra Malik
- Xiaogang Wang
- Andrew Zisserman
In Alignment & RLHF
Citations in the corpus
- Scaling Laws for Reward Model Overoptimization (2022, background): “More broadly, AI alignment is the problem of ensuring that the goals of AI systems are aligned with the goals of humans (Ngo, 2022), including future AI systems which may exceed humans (Bostrom, 2014).”
- Managing extreme AI risks amid rapid progress (2023, background): “…Future AI systems may exhibit failure modes we have so far seen only in theory or lab experiments, such as AI systems taking control over the training reward-provision channels or exploiting weaknesses in our safety objectives and shutdown mechanisms to advance a particular goal 35,56–58 .”
- Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “While none of the features we probed for were especially critical, the Pile subset context neurons could be considered a minimal precursor for situational awareness (Ngo et al., 2023; Carlsmith, 2022), as these could be used to detect whether an input is from the training or test distribution.”
- The rise and potential of large language model based agents: a survey (2025, positive): “Throughout the interaction, humans play a pivotal role by offering guidance or by regulating the safety, legality, and ethical conduct of agents [341,342].”
- Are aligned neural networks adversarially aligned? (2023, background): “Some alignment researchers [Russell, 2019, Bucknall and Dori-Hacohen, 2022, Ngo, 2022, Carl-smith, 2022] believe that sufficiently advanced language models should be aligned to prevent an existential risk [Bostrom, 2013] to humanity: if this were true, an attack that causes such a model to become…”
- Large Language Model Alignment: A Survey (2023, background): “It is widely acknowledged that the key research agendas of AI alignment include outer alignment, inner alignment and interpretability (Hubinger, 2020b; Ngo, 2022; Krakovna, 2022), from a broad perspective.”
- Mechanistic Interpretability for AI Safety - A Review (2024, background): “In contrast, reinforcement learning tends to create agents by default (Casper et al., 2023a; Ngo et al., 2022).”
- Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, neutral): “Additionally, some have raised concerns (Ngo et al., 2024; Casper et al., 2023) that sophisticated LMs trained with human feedback (Christiano et al., 2023) in settings with easy-to-hard domain shift (Burns et al., 2023; Hase et al., 2024) will be misaligned because, in these settings, “overseer…”
- Inverse Scaling: When Bigger Isn't Better (2023, background): “The literature on AI safety suggests possible reasons why initial standard scaling trends in favor of desirable behavior may reverse with sufficient model capabilities (Ngo et al., 2023), which would appear as inverted-U scaling trends (see §5.4 for one possible example).”
- Language Models Represent Space and Time (2023, background): “Better understanding of if and how LLMs model the world is critical for reasoning about the robustness, fairness, and safety of current and future AI systems (Bender et al., 2021; Weidinger et al., 2022; Bommasani et al., 2021; Hendrycks et al., 2023; Ngo et al., 2023).”
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “RL agents have an incentive to seek power when possible to help them accomplish their goals [Turner, 2021, Turner et al., 2019, Turner and Tadepalli, 2022, Ngo, 2022, Krakovna and Kramar, 2023, Ngo, 2022] Versions of this can emerge from the way that RLHF is typically used to finetune LLMs.”
- Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023, background): “, 2023), including potential future risks that AIs might deceive humans in order to accomplish undesirable goals (Ngo et al., 2022).”
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, methodology): “We assume that pretrained base models (or the equivalent in future paradigms) will be highly intelligent but not highly agentic (e.g. will not have long-term goals)—and consequently will not be deceptively aligned (Hubinger et al., 2019; Ngo et al., 2022; Carlsmith, 2023) out-of-the-box.”