Motivated Value Selection for Artificial Agents

By Stuart Armstrong. 2015

53 citations recorded by the source corpus.

Subfields: Alignment & RLHF, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Alignment & RLHF

In Safety, Robustness & Interpretability

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “Crucial components of the reward process that the agent might interfere with include the feedback for the reward model (Armstrong, 2015; Everitt & Hutter, 2018), the observation the reward model uses to determine the current reward (Ring & Orseau, 2011), the code that implements the reward model,…”
Incomplete Contracting and AI Alignment (2018, background): “Strategic incompleteness in reward design may also become relevant in more advanced systems than those we have today [4] if we contemplate sequential reward design with a powerful agent.”
Alignment for Advanced Machine Learning Systems (2020, background): “Soares et al. (2015), Armstrong (2010), and Orseau and Armstrong (2016) study methods for combining objective functions in such a way that the humans have the ability to switch which function an agent is optimizing, but the agent does not have incentives to cause or prevent this switch.”
Value Alignment or Misalignment - What Will Keep Systems Accountable? (2017, background): “Though recent work has explored utility functions (Armstrong 2015) and formulations (Abel, MacGlashan, and Littman 2016) developed speciﬁcally to avoid treacherous manipulation, one frequent argument for IRL is that traditional reinforcement learning is vulnerable to deception on the part of the…”
Avoiding Wireheading with Value Reinforcement Learning (2016, background): “EEP is tightly related to the conservation of expected ethics principle suggested by Armstrong (2015, Eq.”