Superintelligence: Paths, Dangers, Strategies

By Nick Bostrom. 2014

The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species then would come to depend on the actions of the machine superintelligence. But we have one advantage: we get to make the first move. Will it be possible to construct a seed AI or otherwise to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation? To get closer to an answer to this question, we must make our way through a fascinating landscape of topics and considerations. Read the book and learn about oracles, genies, singletons; about boxing methods, tripwires, and mind crime; about humanity's cosmic endowment and differential technological development; indirect normativity, instrumental convergence, whole brain emulation and technology couplings; Malthusian economics and dystopian evolution; artificial intelligence, and biological cognitive enhancement, and collective intelligence. This profoundly ambitious and original book picks its way carefully through a vast tract of forbiddingly difficult intellectual terrain. Yet the writing is so lucid that it somehow makes it all seem easy. After an utterly engrossing journey that takes us to the frontiers of thinking about the human condition and the future of intelligent life, we find in Nick Bostrom's work nothing less than a reconceptualization of the essential task of our time.

Subfields: Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Alignment & RLHF

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, background): “This difﬁculty underlies recent concerns about misalignment between our values and the objectives of our RL systems (Bostrom, 2014; Russell, 2016; Amodei et al., 2016).”
When Will AI Exceed Human Performance? Evidence from AI Experts (2017, background): “Some authors have argued that once HLMI is achieved, AI systems will quickly become vastly superior to humans in all tasks [3, 13].”
Artificial Intelligence, Values, and Alignment (2020, background): “Indeed, this tendency sits at the heart of certain long-term safety concerns about AI; if it were to optimise for something that we did not really want, then this could have serious consequences for the world (Bostrom 2016).”
Risks from Learned Optimization in Advanced Machine Learning Systems (2019, background): “In its most extreme form, this failure of alignment could lead to a “treacherous turn” as described by Nick Bostrom in Superintelligence.”
The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities (2018, neutral): “…therein are concerned with the potential for perverse outcomes from optimizing reward functions that appear sensible on their surface [4, 11, 37, 134], characterized in that community as problems such as avoiding negative side effects [4, 69], reward hacking (also known as wire-heading)…”
Categorizing Variants of Goodhart's Law (2018, background): “This discussion should be helpful in better understanding these types of failures in economic regulation, in public policy, in machine learning, and in artificial intelligence alignment[3].”
AI Safety Gridworlds (2017, background): “Expecting that more advanced versions of today’s AI systems are going to be deployed in realworld applications, numerous public figures have advocated more research into the safety of these systems (Bostrom, 2014; Hawking et al., 2014; Russell, 2016).”
Modeling AGI Safety Frameworks with Causal Influence Diagrams (2019, background): “Although the development of AGI has many potential benefits, there are also many safety concerns that have been raised in the literature [Bostrom, 2014; Everitt et al., 2018; Amodei et al., 2016].”
The Forthcoming Artificial Intelligence (AI) Revolution: Its Impact on Society and Firms (2017, background): “Joy and many other scientists (Cellan-Jones, 2016) and philosophers (Bostrom, 2014) believe that Kurzweil and his supporters vastly underestimate the magnitude of the challenge and the potential dangers which can arise from thinking machines and intelligent robots.”
Strategic Implications of Openness in AI Development (2017, unknown): “Another concern is that in the absence of global coordination it may be impossible to forestall a population explosion of digital minds and a resulting Malthusian era in which the welfare of those digital minds may suffer (Bostrom, 2004, 2014a; Hanson, 1994).”
FHI Oxford Technical Report # 2018-2 Predicting Human Deliberative Judgments with Machine Learning (2018, background): “the problem of creating AI systems that remain aligned with human preferences even as their capabilities exceed those of humans [17, 18, 19, 20, 21].”
Motivated Value Selection for Artificial Agents (2015, background): “The concept of utility indifference (Armstrong 2010) grew out originally of a problem in controlling a (putative) artificial intelligence (Bostrom 2014; Yudkowsky 2008).”
The Challenges of Algorithm-Based HR Decision-Making for Personal Integrity (2019, background): “, the point at which machines become too smart for humankind to maintain control over its own fate (Boström 2014).”
Safely Interruptible Agents (2016, background): “Some people have also expressed concerns that a “superin-telligent” agent may resist being shut down, because this would lead to a decrease of its expected reward [Omo-hundro, 2008, Bostrom, 2014].”
Artificial intelligence in business: State of the art and future research agenda (2020, background): “The integration of neurostimulators and nanochips into the brain enhances the human organism toward transhumanism (Bostrom, 2005a; Bostrom, 2014), and eventual creation of the hybrid humans (e.g., Kumar, Dixit, Javalgi, & Das, 2016; Rust & Kannan, 2002; Ng & Wakenshaw, 2017).”
The Ethics of AI Ethics: An Evaluation of Guidelines (2019, background): “ost striking aspect is that the danger of a malevolent AGI or existential threats is not discussed in any of the ethics guidelines, even though this is a thoroughly discussed topic in other contexts (Bostrom 2014; Tegmark 2017; Müller and Bostrom 2016; Omohundro 2014). This may be due to the fact th”
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective (2019, background): “ other problems that we have considered in this paper, as they will usually render the agent ineﬀective. Therefore, the risk that cheating-the-world-model would lead to e.g. infrastructure profusion (Bostrom, 2014) may be smaller than for the other problems, in which the agent often remains capable ”
Artificial Intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy (2019, background): “…types of arguments (for a concise review of the arguments see Eden, Steinhart, Pearce, & Moor, 2012; for their most recent instalment, Bostrom, 2014) has become relatively fashionable (as I often say in my talks “ unhype is the new hype ” ), it is time to strategically intervene and…”