Alignment for Advanced Machine Learning Systems

By Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, Andrew Critch. 2020

95 citations recorded by the source corpus.

This chapter surveys eight research areas organized around one question: As learning systems become increasingly intelligent and autonomous, what design principles can best ensure that their behavior is aligned with the interests of the operators? The chapter focuses on two major technical obstacles to AI alignment: the challenge of specifying the right kind of objective functions and the challenge of designing AI systems that avoid unintended consequences and undesirable behavior even in cases where the objective function does not line up perfectly with the intentions of the designers. The questions surveyed include the following: How can we train reinforcement learners to take actions that are more amenable to meaningful assessment by intelligent overseers? What kinds of objective functions incentivize a system to “not have an overly large impact” or “not have many side effects”? The chapter discusses these questions, related work, and potential directions for future research, with the goal of highlighting relevant research topics in machine learning that appear tractable today.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “…problem deﬁnitions from many authors before us, highlighting tractable and neglected problems in the ﬁeld of AI safety (Russell et al., 2015; Soares, 2015; Amodei et al., 2016; Taylor et al., 2016; Soares & Fallenstein, 2017; Christiano, 2017; Leike et al., 2017; Ortega et al., 2018; and others).”
Unsolved Problems in ML Safety (2021, background): “, 2016 [184] considers similar topics to Amodei and Olah et al.”
Trust in AI: progress, challenges, and future directions (2024, unknown): “…of (1) making sure that an arti ﬁ cial generalized intelligence system has values and interests aligned with those of humans (Abbass, 2019c; Bostrom, 2014; Gabriel, 2020; Prasad, 2019; Taylor et al., 2016) and (2) determining how we could be assured that we succeeded in ensuring this alignment.”
Measuring and avoiding side effects using relative reachability (2018, background): “, 2017) and low impact (Armstrong and Levinstein, 2017; Taylor et al., 2016), use criteria of this form.”
AGI Safety Literature Review (2018, methodology): “More up-to-date references are provided by this chapter, and by a number of recent research agendas and problem collections (Amodei, Olah, et al., 2016; Leike, Martic, et al., 2017; Russell et al., 2016; Soares and Fallenstein, 2017; Stoica et al., 2017; Taylor et al., 2016).”
FHI Oxford Technical Report # 2018-2 Predicting Human Deliberative Judgments with Machine Learning (2018, background): “So while we cannot hope for highly accurate predictions of slow judgments, we can seek ML algorithms that “know what they know” [44].”
Penalizing Side Effects using Stepwise Relative Reachability (2018, background): “An important component of safe behavior for reinforcement learning agents is avoiding unnecessary side effects while performing a task [Amodei et al., 2016; Taylor et al., 2016].”
Towards Safe Artificial General Intelligence (2018, methodology): “For brevity, we will refer to the agendas by the organization of the first author, with MIRI-AF the agent foundations agenda by Soares and Fallenstein (2014, 2017) and MIRI-ML the machine learning agenda by (Taylor et al., 2016).”