Aligning Superintelligence with Human Interests: A Technical Research Agenda

By Nate Soares, Benja Fallenstein. 2015

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

Adversarial Training for High-Stakes Reliability (2022, background): “Most empirical work on techniques for aligning powerful AI [12, 13, 14, 15, 16] has focused on achieving good average-case performance in domains where no single action is catastrophic, for example using human trajectory rankings [17, 18, 19] or imitation learning [20, 21].”
Research Priorities for Robust and Beneficial Artificial Intelligence (2015, background): “Explicitly specifying our preferences in broad domains in the style of near-future machine ethics may not be practical, making “aligning” the values of powerful AI systems with our own values and preferences diﬃcult [70, 71].”
AI Safety Gridworlds (2017, background): “This nascent field of AI safety still lacks a general consensus on its research problems, and there have been several recent efforts to turn these concerns into technical problems on which we can make direct progress (Soares and Fallenstein, 2014; Russell et al., 2015; Taylor et al., 2016; Amodei et”
Questions of Reasoning Under Logical Uncertainty (2015, background): “If high confidence in this reasoning is to be justified, even in a wide array of esoteric situations, then a theoretical understanding of logically uncertain reasoning is necessary: without it, it is difficult to ask the right questions (Soares and Fallenstein 2014a).”
Reducing Non-Normative Text Generation from Language Models (2020, background): “This is referred to as the value alignment problem 1 https://www.newsweek.com/amazon-echo% 2Dtells-uk-woman-stab-herself-1479074/ (Soares and Fallenstein, 2014; Russell et al., 2015; Arnold et al., 2017; Abel et al., 2016).”