The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents

By Nick Bostrom. 2012

259 citations recorded by the source corpus.

Subfields: Computational Theory and Mathematics, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Information Systems

Citations in the corpus

Research Priorities for Robust and Beneficial Artificial Intelligence (2015, background): “If an AI system is selecting the actions that best allow it to complete a given task, then avoiding conditions that prevent the system from continuing to pursue the task is a natural subgoal (Omohundro 2007; Bostrom 2012) (and conversely, seeking unconstrained situations is sometimes a useful heuris”
The alignment problem from a deep learning perspective (2022, background): “The core intuition underlying concerns about power-seeking is Bostrom [2012]’s instrumental convergence thesis , which states that there are some subgoals that are instrumentally useful for achieving almost any ﬁnal goal.”
Ethics of artificial intelligence and robotics (2020, background): “Arguments for existential risk from superintelligence typically deny this and say that rationality and morality are entirely independent or “orthogonal” dimensions – this is sometimes explicitly argued for as an “orthogonality thesis” (Bostrom 2012; Armstrong 2013; Bostrom 2014, 105-109).”
Large Language Model Alignment: A Survey (2023, background): “Among these, the orthogonality thesis (OT) (Bostrom, 2012) and instrumental convergence thesis (ICT) (Omohundro, 2008; Bostrom, 2012; Armstrong et al.”
Optimal Policies Tend To Seek Power (2019, unknown): “…in the sense that their attainment would increase the chances of the agent’s goal being realized for a wide range of ﬁnal goals and a wide range of situations, implying that these instrumental values are likely to be pursued by a broad spectrum of situated intelligent agents [Bostrom, 2012].”
AGI Safety Literature Review (2018, background): “In stark contrast, the instrumental goals will often coincide for many agents and end goals (Bostrom, 2012; Omohundro, 2007, 2008).”
When Combating Hype, Proceed with Caution (2021, background): “tems that are optimizing for benign objectives, once they become sufficiently capable, have a predictable reason to take on dangerous subgoals— like accumulating large amounts of computational, economic, or political power—to maximize the odds that their primary objectives are achieved (Bostrom, 200”
Thinking Inside the Box: Controlling and Using an Oracle AI (2012, background): “This is an extreme form of ''mind crime'' (Bostrom 2012a) where the OAI causes great destruction just by thinking.”
Motivated Value Selection for Artificial Agents (2015, background): “But an agent could have any conceivable motivational structure (Bostrom 2012; Armstrong 2013), ignoring the ‘true morality’ even if such a thing existed.”
Defining and Characterizing Reward Hacking (2022, unknown): “A concern more specific to the optimization of reward functions is power-seeking (Turner et al., 2021; Bostrom, 2012; Omohundro, 2008).”
Corrigibility (2015, background): “…regard as erroneous goals would have an incentive to resist being corrected: general analysis of rational agents has suggested that almost all such agents are instrumentally motivated to preserve their preferences, and hence to resist attempts to modify them (Bostrom 2012; Yudkowsky 2008).”
Formalizing Convergent Instrumental Goals (2016, background): “Bostrom [4] refines this into the “instrumental convergence” thesis, which states that certain instrumentally useful goals will likely be pursued by a broad spectrum of intelligent agents—such goals are said to be “convergent instrumental goals.””