The Value Learning Problem

By Nate Soares. 2018

86 citations recorded by the source corpus.

Autonomous AI systems’ programmed goals can easily fall short of programmers’ intentions. Even a machine intelligent enough to understand its de-signers’ intentions would not necessarily act as intended. We discuss early ideas on how one might design smarter-than-human AI systems that can inductively learn what to value from labeled training data, and highlight questions about the construction of systems that model and act upon their operators’ preferences.

Subfields: Artificial Intelligence, Safety Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “…and problem deﬁnitions from many authors before us, highlighting tractable and neglected problems in the ﬁeld of AI safety (Russell et al., 2015; Soares, 2015; Amodei et al., 2016; Taylor et al., 2016; Soares & Fallenstein, 2017; Christiano, 2017; Leike et al., 2017; Ortega et al., 2018; and…”
Self-Modification of Policy and Utility Function in Rational Agents (2016, background): “Examples include natural variants of value learning (Dewey, 2011; Soares, 2015), corrigibility (Soares et al.”
Avoiding Wireheading with Value Reinforcement Learning (2016, background): “In the extreme case, the agent would even self-modify to remove its learning ability (Everitt et al., 2016; Soares, 2015).”
Value Alignment or Misalignment - What Will Keep Systems Accountable? (2017, background): “Some advocates of alignment have pointed out that reliance on observation alone may not capture the ethical character of human action (Soares 2015).”
Formalizing Convergent Instrumental Goals (2016, background): “Armstrong (2010) and Soares et al. (2015) have done some initial study into the design of goals which robustly avert certain convergent instrumental incentives.”
Quantilizers: A Safer Alternative to Maximizers for Limited Optimization (2016, background): “More formally, say we desire some distribution p ∈ ∆A such that the following conservative cost constraint holds for some acceptable cost threshold t > 1: ∀c :A → R≥0 EA∼γ [c(A)] ≤ 1 =⇒ EA∼p [c(A)] ≤ t. (2) This constraint states that, for any cost function assigning an expected cost of no more…”
Using Stories to Teach Human Values to Artificial Agents (2016, background): “Further, the preferences of humans many not necessarily be easily inferred by observations (Soares 2015).”
Alignment for Advanced Machine Learning Systems (2020, background): “To achieve high conﬁdence in value learning systems, however, Soares (2016) argues that we will need to be able to anticipate cases where the system’s past experiences of preferred and unpreferred outcomes provide insuﬃcient evidence for inferring whether future outcomes are desirable.”