Optimal Policies Tend To Seek Power

By A. Turner, Logan Smith, Rohin Shah, Andrew Critch, Prasad Tadepalli. 2019

130 citations recorded by the source corpus.

Some researchers speculate that intelligent reinforcement learning (RL) agents would be incentivized to seek resources and power in pursuit of their objectives. Other researchers point out that RL agents need not have human-like power-seeking instincts. To clarify this discussion, we develop the first formal theory of the statistical tendencies of optimal policies. In the context of Markov decision processes, we prove that certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment. These symmetries exist in many environments in which the agent can be shut down or destroyed. We prove that in these environments, most reward functions make it optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Scaling Laws for Reward Model Overoptimization (2022, background): “There are a number of reasons to expect AI misalignment, especially in those more powerful future systems, to occur [Omohundro, 2008, Turner et al., 2021, Armstrong et al., 2013, Hubinger et al., 2019, Soares et al., 2015], and to result in catastrophic outcomes [Carlsmith, 2022, Cotra, 2022].”
Unsolved Problems in ML Safety (2021, background): “Sometimes unintended instrumental goals emerge in systems, such as self-preservation [69] or power-seeking [191], so researchers could try mitigating and detecting such unintended emergent goals; see Section 3.3.2 for more directions in detecting emergent functionality.”
Objective Robustness in Deep Reinforcement Learning (2021, background): “For example, for most reward functions, the optimal policy will try to avoid being shut down [35].”
Large Language Model Alignment: A Survey (2023, background): “Existing studies (Turner et al., 2021; Turner and Tadepalli, 2022; Krakovna and Kramar, 2023) have demonstrated that optimal polices and reward functions may incentivize systems to pursue power in certain environments.”
The alignment problem from a deep learning perspective (2022, background): “A formal statement of the instrumental convergence thesis is provided by Turner et al. [2021], who deﬁne a state’s “power” as its average value across a wide range of reward functions.”
Managing extreme AI risks amid rapid progress (2023, background): “To advance undesirable goals, future autonomous AI systems could use undesirable strategies—learned from humans or developed independently—as a means to an end 32–35 .”
Defining and Characterizing Reward Hacking (2022, unknown): “…result only applies to environments with decreasing marginal utility and increasing opportunity cost, we demonstrate hackability is an issue in arbitrary MDPs. Hackability is particularly concerning given arguments that reward optimizing behavior tends to be power-seeking (Turner et al., 2021).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “RL agents have an incentive to seek power when possible to help them accomplish their goals [Turner, 2021, Turner et al., 2019, Turner and Tadepalli, 2022, Ngo, 2022, Krakovna and Kramar, 2023, Ngo, 2022] Versions of this can emerge from the way that RLHF is typically used to finetune LLMs.”