Towards Safe Artificial General Intelligence

By Tom Everitt. 2018

24 citations recorded by the source corpus.

Subfields: Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

1. Tom Everitt
2. Shane Legg
3. Victoria Krakovna
4. Miljan Martic
5. Jan Leike
6. Ramana Kumar
7. David Krueger
8. Vishal Maini

In Signal Processing

1. Shane Legg
2. Jan Leike
3. Ramana Kumar
4. Victoria Krakovna
5. David Krueger
6. Miljan Martic
7. Vishal Maini
8. Dario Amodei

Citations in the corpus

• Scalable agent alignment via reward modeling: a research direction (2018, methodology): “Agents that plan this way have no incentive to change their reward functions (Schmidhuber, 2007; Omohundro, 2008), nor manipulate the register holding the reward signal (Everitt, 2018, Sec.”
• Reward learning from human preferences and demonstrations in Atari (2018, background): “This so-called reward hacking problem (Amodei et al., 2016; Everitt, 2018) is not unique to reward learning; hard-coded reward functions are also exploitable in this way (Lehman et al.”
• Modeling AGI Safety Frameworks with Causal Influence Diagrams (2019, methodology): “An elegant solution to this problem is to use model-based agents that simulate the state sequence likely to result from different policies, and evaluate those state sequences according to the current or initial reward function [Everitt, 2018; Orseau and Ring, 2011; Hibbard, 2012; Schmidhuber, 2007; ”