Should Robots be Obedient?

By Smitha Milli, Dylan Hadfield-Menell, Anca Dragan, Stuart J. Russell. 2017

70 citations recorded by the source corpus.

Intuitively, obedience -- following the order that a human gives -- seems like a good property for a robot to have. But, we humans are not perfect and we may give orders that are not best aligned to our preferences. We show that when a human is not perfectly rational then a robot that tries to infer and act according to the human's underlying preferences can always perform better than a robot that simply follows the human's literal order. Thus, there is a tradeoff between the obedience of a robot and the value it can attain for its owner. We investigate how this tradeoff is impacted by the way the robot infers the human's preferences, showing that some methods err more on the side of obedience than others. We then analyze how performance degrades when the robot has a misspecified model of the features that the human cares about or the level of rationality of the human. Finally, we study how robots can start detecting such model misspecification. Overall, our work suggests that there might be a middle ground in which robots intelligently decide when to obey human orders, but err on the side of obedience.

Subfields: Artificial Intelligence, Management of Technology and Innovation, Social Psychology, AI Ethics, Fairness & Society, Robotics & Control, Philosophy

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Social Psychology

In AI Ethics, Fairness & Society

In Robotics & Control

In Philosophy

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, neutral): “When should the agent be disobedient (Milli et al., 2017)?”
Machine Theory of Mind (2018, background): “They will also be highly useful for communication and pedagogy (Dragan et al., 2013; Fisac et al., 2017; Milli et al., 2017), and will thus likely play a key role in human-machine interaction.”
Occam's razor is insufficient to infer the preferences of irrational agents (2017, background): “Staying close to agnostic is desirable in some settings: for example, a misspecified model of the human reward function can lead to disastrous decisions with high confidence [Milli et al., 2017].”
AI Safety Gridworlds (2017, background): “d. Other proposed solutions include continuing the episode in simulation upon interruption (Riedl and Harrison, 2017) and retaining uncertainty over the reward function (Hadﬁeld-Menell et al., 2016b; Milli et al., 2017). 2.1.2 Avoiding side effects When we ask an agent to achieve a goal, we usually ”
Optimal Policies Tend To Seek Power (2019, background): “Many AI alignment researchers hypothesize that most advanced AI agents will have concerning instrumental incentives, such as resisting deactivation [Soares et al., 2015, Milli et al., 2017, Hadﬁeld-Menell et al., 2017, Carey, 2018] and acquiring resources [Benson-Tilsen and Soares, 2016].”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, background): “Research has shown that difﬁculties arise when an agent seeks to align its value with a human who is not perfectly rational (Milli et al., 2017) and that there are fundamental impossibility results regarding value alignment (Eckersley, 2018); however, Eckersley (2018) demonstrate that these…”
AGI Safety Literature Review (2018, methodology): “There has been some debate about whether this is a feature (Milli et al., 2017) or a bug (Carey, 2018).”