Formalizing Convergent Instrumental Goals

By Tsvi Benson-Tilsen, Nate Soares. 2016

36 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Tom Everitt
2. Andrew Critch
3. Jessica Taylor
4. Patrick LaVictoire
5. Eliezer Yudkowsky
6. A. Turner
7. Rohin Shah
8. Logan Smith

Citations in the corpus

• Optimal Policies Tend To Seek Power (2019, background): “Many AI alignment researchers hypothesize that most advanced AI agents will have concerning instrumental incentives, such as resisting deactivation [Soares et al., 2015, Milli et al., 2017, Hadfield-Menell et al., 2017, Carey, 2018] and acquiring resources [Benson-Tilsen and Soares, 2016].”
• Alignment for Advanced Machine Learning Systems (2020, background): “It is difficult to define practical objective functions that resist these pressures (Benson-Tilsen and Soares 2016).”