Ethical Issues in Advanced Artificial Intelligence

By Nick Bostrom. 2020

183 citations recorded by the source corpus.

The ethical issues related to the possible future creation of machines with general intellectual capabilities far outstripping those of humans are quite distinct from any ethical problems arising in current automation and information systems. Such superintelligence would not be just another technological development; it would be the most important invention ever made, and would lead to explosive progress in all scientific and technological fields, as the superintelligence would conduct research with superhuman efficiency. To the extent that ethics is a cognitive pursuit, a superintelligence could also easily surpass humans in the quality of its moral thinking. However, it would be up to the designers of the superintelligence to specify its original motivations. Since the superintelligence may become unstoppably powerful because of its intellectual superiority and the technologies it could develop, it is crucial that it be provided with human-friendly motivations. This paper surveys some of the unique ethical issues in creating superintelligence, and discusses what motivations we ought to give a superintelligence, and introduces some cost-benefit considerations relating to whether the development of superintelligent machines ought to be accelerated or retarded.

Subfields: Astronomy and Astrophysics, Cognitive Neuroscience, Safety Research, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Astronomy and Astrophysics

1. Tom Everitt
2. Peter Eckersley
3. Samuel R. Bowman

In Cognitive Neuroscience

1. Tom Everitt
2. David Krueger
3. Peter Eckersley
4. Miljan Martic
5. Jan Leike
6. Shane Legg
7. Samuel R. Bowman
8. Ben Goertzel

In Safety Research

1. Shane Legg
2. Tom Everitt
3. Peter Eckersley
4. Miljan Martic
5. Samuel R. Bowman
6. Iason Gabriel

In AI Ethics, Fairness & Society

1. Tom Everitt
2. Peter Eckersley
3. Miljan Martic
4. Shane Legg
5. Samuel R. Bowman

Citations in the corpus

• Scalable agent alignment via reward modeling: a research direction (2018, background): “Since then, the agent alignment problem has been echoed by philosophers (Bostrom, 2003, 2014; Yudkowsky, 2004) and treated informally by technical authors (Wiener, 1960; Etzioni & Weld, 1994; Omohundro, 2008).”
• What Question Answering can Learn from Trivia Nerds (2019, unknown): “blindly optimize an objective function (Bostrom, 2003).”
• When Combating Hype, Proceed with Caution (2021, background): “tems that are optimizing for benign objectives, once they become sufficiently capable, have a predictable reason to take on dangerous subgoals— like accumulating large amounts of computational, economic, or political power—to maximize the odds that their primary objectives are achieved (Bostrom, 200”
• Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function) (2018, background): “…Many of the concerns in the literature about the difficulty of aligning hypothetical future AGI systems to human values are motivated by the risk of “instrumental convergence” of those systems — the adoption of sub-goals that are dis-empowering of humans and other agents [37, 9, 32, 44, 10, 42].”
• Artificial Intelligence and Marketing: Pitfalls and Opportunities (2020, background): “In his canonical thought experiment, Bostrom (2003) develops the case where an artificial general intelligence tasked with the objective of manufacturing as many paperclips as possible, although designed without malice, could ultimately destroy humanity since no reward for humanity's terminal…”