Superintelligence: Paths, Dangers, Strategies

By Christian P. Robert. 2017

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Alignment & RLHF

Citations in the corpus

Ethical and social risks of harm from Language Models (2021, background): “This report also does not cover risks that depend on superintelligence as described in (Bostrom, 2014).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “Alignment is, in a sense, “philosophy with a deadline” [Bostrom, 2014].”
WebGPT: Browser-assisted question-answering with human feedback (2021, background): “Nevertheless, much more capable models could potentially pose much more serious risks [Bostrom, 2014].”
What can the brain teach us about building artificial intelligence? (2016, methodology): “One reason humans might be able to learn from fewer games than a neural network trained on playing chess as a single integrated task is that humans receive feedback on many of these tasks throughout learning, and this both allows more feedback from a single experience (e.g. both an emotional reward ”
Alignment of Language Agents (2021, background): “In this paper we focus on behavioural issues that arise due to a lack of alignment, where the system does not do what we intended it to do (Bostrom, 2014; Christiano, 2018; Leike et al., 2018; Russell, 2019).”
Supervising strong learners by amplifying weak experts (2018, background): “At large scales this kind of pathology could lead to systemic crashes, and a mismatch between proxies and our real preferences is a major source of concerns about the safety of future powerful AI systems (Bostrom, 2014).”
Scaling Laws for Reward Model Overoptimization (2022, background): “More broadly, AI alignment is the problem of ensuring that the goals of AI systems are aligned with the goals of humans [Ngo, 2022], including future AI systems which may exceed humans [Bostrom, 2014].”
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2022, background): “While we do not ask how specifically respondents think this could happen, potential reasons for such concerns are discussed by Bostrom (2014); Amodei et al. (2016) and Hubinger et al. (2019).”
Adversarial Training for High-Stakes Reliability (2022, methodology): “Contractors took on average 20 minutes (95% CI: [16, 24]) to ﬁnd an adversarial example for the baseline classiﬁer without the tool, compared to 13 minutes with the tool (95% CI: [12, 14]).”
Aligning AI With Shared Human Values (2020, background): “The final scenario refers to Bostrom (2014)’s paperclip maximizer.”
Research Priorities for Robust and Beneficial Artificial Intelligence (2015, background): “Potential research topics in this area include “domestic” goals that are limited in scope in some way [72], the eﬀects of large temporal discount rates on resource acquisition strategies, and experimental investigation of simple systems that display these subgoals.”
Cooperative Inverse Reinforcement Learning (2016, background): “A solution to the value alignment problem has long-term implications for the future of AI and its relationship to humanity (Bostrom, 2014) and short-term utility for the design of usable AI systems.”
Occam's razor is insufficient to infer the preferences of irrational agents (2017, background): “Other authors have argued directly for the complexity of human values, from ﬁelds as diverse as computer science, philosophy, neuroscience, and economics [Minsky, 1984, Bostrom, 2014b, Glimcher et al., 2009, Muehlhauser and Helm, 2012, Yudkowsky, 2011].”
Reinforcement Learning with a Corrupted Reward Channel (2017, background): “Reward corruption may impede the performance of a wide range of agents, and may have disastrous consequences for highly intelligent agents [Bostrom, 2014].”
Taxonomy of Risks posed by Language Models (2022, background): “While many of these risks intersect with longer-term concerns, we refer to other work [26, 53, 99] for more focused discussion of risks in the context of AGI safety and superintelligence.”
Avoiding Wireheading with Value Reinforcement Learning (2016, background): “Other value learning suggestions have been speculative or vague (Bostrom, 2014a,b; Dewey, 2011).”