Risks from Learned Optimization in Advanced Machine Learning Systems

By Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Max Viktor Skalse, Scott Garrabrant. 2019

297 citations recorded by the source corpus.

We analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer - a situation we refer to as mesa-optimization, a neologism we introduce in this paper. We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems. First, under what circumstances will learned models be optimizers, including when they should not be? Second, when a learned model is an optimizer, what will its objective be - how will it differ from the loss function it was trained under - and how can it be aligned? In this paper, we provide an in-depth analysis of these two primary questions and provide an overview of topics for future research.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “Being able to interpret and inspect the detailed inner workings of trained models would be a powerful tool to ensure models are optimizing for the goals we intended (Hubinger et al., 2021; Koch et al., 2021).”
Alignment of Language Agents (2021, methodology): “Hubinger et al. (2019) introduce the concept of a mesa-optimizer – a learnt model which is itself an optimizer for some mesa-objective, which may differ from the base-objective used to train the model, when deployed outside of the training environment.”
Measuring Progress on Scalable Oversight for Large Language Models (2022, background): “…that could be elicited with the help of additional incentives or model inter-pretability techniques (see e.g. Christiano et al., 2021), and it is plausible that eliciting information of this kind will be important to the successful oversight of future highly-capable systems (Hubinger et al., 2019).”
Scaling Laws for Reward Model Overoptimization (2022, background): “There are a number of reasons to expect AI misalignment, especially in those more powerful future systems, to occur [Omohundro, 2008, Turner et al., 2021, Armstrong et al., 2013, Hubinger et al., 2019, Soares et al., 2015], and to result in catastrophic outcomes [Carlsmith, 2022, Cotra, 2022].”
Discovering Language Model Behaviors with Model-Written Evaluations (2022, background): “Alternatively, such models may intentionally obtain high reward during training, in order to later be deployed and carry out undesirable plans (Hubinger et al., 2019).”
Adversarial Training for High-Stakes Reliability (2022, background): “(2)Either situation could be caused by deceptive alignment [23]: a system might calibrate its probability of misbehaving to be unlikely in training but likely during deployment (leading to failures in-distribution), or it might wait until its inputs make it clear that the system is actually deployed”
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “…that better algorithms and models are not enough; for human-centered applications such as healthcare (Yu et al., 2019), economics (Trott et al., 2021) and robotics (Kober et al., 2013), RL algorithms must be safe and aligned with human objectives (Bommasani et al., 2021; Hubinger et al., 2019).”
Unsolved Problems in ML Safety (2021, background): “Robustness work could also move beyond classiﬁcation and consider competent errors where agents misgeneralize and execute wrong routines, such as an automated digital assistant knowing how to use a credit card to book ﬂights, but choosing the wrong destination [101, 91].”
Inverse Scaling: When Bigger Isn't Better (2023, background): “One particularly important, potential emergent risk is deceptive alignment (Hubinger et al., 2021): an AI system that appears to pursue a given objective under the training distribution but pursues an alternative objective off-distribution.”
The alignment problem from a deep learning perspective (2022, methodology): “The problem of ensuring that policies learn desirable internally-represented goals is known as the inner alignment problem , in contrast to the “outer” alignment problem of providing well-speciﬁed rewards [Hubinger et al., 2021].”
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, methodology): “…to primarily theoretical frameworks and toy problems (Irving et al., 2018; Chris-tiano et al., 2018; Leike et al., 2018; Demski & Garrabrant, 2019; Hubinger et al., 2019), or empirically studied humans supervising today’s models—without addressing the core challenges that may arise with…”
Transformers learn in-context by gradient descent (2022, background): “Recently, this concept of an emergent optimizer within a learned neural network, such as a Transformer, has been termed “mesa-optimization” (Hubinger et al., 2019).”
Objective Robustness in Deep Reinforcement Learning (2021, background): “[18], we (non-exhaustively) list factors that increase the likelihood of learning a particular proxy R′.”
When Combating Hype, Proceed with Caution (2021, background): “Further, optimizing complex systems is difﬁcult, such that even a genuinely safe objective, if optimized powerfully by a system with an imperfect learned representation of the objective, could produce dangerous failure modes (Hubinger et al., 2019).”
Ignore Previous Prompt: Attack Techniques For Language Models (2022, unknown): “Regarding Q7, we have intentionally modeled our heuristics after human-level content sophistication.”