The Basic AI Drives

By S. Omohundro. 2008

396 citations recorded by the source corpus.

One might imagine that AI systems with harmless goals will be harmless. This paper instead shows that intelligent systems will need to be carefully designed to prevent them from behaving in harmful ways. We identify a number of “drives” that will appear in sufficiently advanced AI systems of any design. We call them drives because they are tendencies which will be present unless explicitly counteracted. We start by showing that goal-seeking systems will have drives to model their own operation and to improve themselves. We then show that self-improving systems will be driven to clarify their goals and represent them as economic utility functions. They will also strive for their actions to approximate rational economic behavior. This will lead almost all systems to protect their utility functions from modification and their utility measurement systems from corruption. We also discuss some exceptional systems which will want to modify their utility functions. We next discuss the drive toward self-protection which causes systems try to prevent themselves from being harmed. Finally we examine drives toward the acquisition of resources and toward their efficient utilization. We end with a discussion of how to incorporate these insights in designing intelligent technology which will lead to a positive future for humanity.

Subfields: Economics and Econometrics, Safety, Robustness & Interpretability, Philosophy

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Safety, Robustness & Interpretability

In Philosophy

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “Agents that plan this way have no incentive to change their reward functions (Schmidhuber, 2007; Omohundro, 2008), nor manipulate the register holding the reward signal (Everitt, 2018, Sec.”
Scaling Laws for Reward Model Overoptimization (2022, background): “There are a number of reasons to expect AI misalignment, especially in those more powerful future systems, to occur [Omohundro, 2008, Turner et al., 2021, Armstrong et al., 2013, Hubinger et al., 2019, Soares et al., 2015], and to result in catastrophic outcomes [Carlsmith, 2022, Cotra, 2022].”
Discovering Language Model Behaviors with Model-Written Evaluations (2022, background): “Long-term plans may cause systems to alter their environment in drastic or unexpected ways (Omohundro, 2008; Cotra, 2021a), e.”
AI Safety Gridworlds (2017, background): “rcome robustness problems: if the agent could ﬁnd a way to be more robust, it would likely gather more reward. As such, robustness can be seen as a subgoal or instrumental goal of intelligent agents (Omohundro, 2008; Bostrom, 2014, Ch. 7). In contrast, speciﬁcation problems do not share this self-co”
Artificial Intelligence and Its Implications for Income Distribution and Unemployment (2017, neutral): “…of the likely characteristics of any sufficiently intelligent entity – no matter what final objectives are programmed into it by evolution or by its creator – is that it will act by pursuing intermediate objectives or “basic drives” that are instrumental for any final objective (Omohundro, 2008).”
Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function) (2018, background): “Many of the concerns in the literature about the difficulty of aligning hypothetical future AGI systems to human values are motivated by the risk of “instrumental convergence” of those systems — the adoption of sub-goals that are disempowering of humans and other agents (Russell and Norvig 2003; Bos”
Corrigibility (2015, background): “In Stephen Omo-hundro’s terms, “goal-content integrity” is an instrumentally convergent goal of almost all intelligent agents (Omohundro 2008).”
Good and safe uses of AI Oracles (2017, background): “However there are arguments that any program, especially if capable of self-modification, may change itself into an agent that is an expected utility maximiser [Omohundro, 2008].”
Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda (2017, background): “However, nearly all goals can be better met with more resources (Omohundro 2008).”
Self-Modeling Agents and Reward Generator Corruption (2015, background): “Unintended instrument actions: Omohundro described actions (he called them basic drives) of future AI systems that will be instrumental to a wide range of goals they may be designed to achieve (Omohundro 2008).”
Incomplete Contracting and AI Alignment (2018, neutral): “ts human designer share, in principle, the same goal. That goal is set by the human designer. An AI agent, it would seem, is not a strategic actor that needs to be incentivized to behave well. But as Omohundro (2008) ﬁrst observed, an AI’s goal is established by its stated reward function and this m”
Embedded Agency (2019, background): “The problem is then that powerful inner optimizers are optimizing something the outer system doesn’t care about, and that the inner optimizers will have a convergent incentive to be agentic (Branwen 2016; Omohundro 2008).”
The Singularity: a Philosophical Analysis (2010, background): “See also Omohundro (2007, 2008). better species, it is possible to see the moral calculus as going either way.”
The Off-Switch Game (2016, background): “[Omohundro, 2008] considers instrumental goals of artificial agents: goals which are likely to be adopted as subgoals of most objectives.”