Learning the Preferences of Ignorant, Inconsistent Agents

By Owain Evans, Andreas Stuhlmüller, Noah D. Goodman. 2015

138 citations recorded by the source corpus.

An important use of machine learning is to learn what people value. What posts or photos should a user be shown? Which jobs or activities would a person find rewarding? In each case, observations of people's past choices can inform our inferences about their likes and preferences. If we assume that choices are approximately optimal according to some utility function, we can treat preference inference as Bayesian inverse planning. That is, given a prior on utility functions and some observed choices, we invert an optimal decision-making process to infer a posterior distribution on utility functions. However, people often deviate from approximate optimality. They have false beliefs, their planning is sub-optimal, and their choices may be temporally inconsistent due to hyperbolic discounting and other biases. We demonstrate how to incorporate these deviations into algorithms for preference inference by constructing generative models of planning for agents who are subject to false beliefs and time inconsistency. We explore the inferences these models make about preferences, beliefs, and biases. We present a behavioral experiment in which human subjects perform preference inference given the same observations of choices as our model. Results show that human subjects (like our model) explain choices in terms of systematic deviations from optimal behavior and suggest that they take such deviations into account when inferring preferences.

Subfields: Artificial Intelligence, General Decision Sciences, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Decision Sciences

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “Even some assumptions about the human’s rationality can be relaxed (Evans et al., 2016), but in full generality the inverse reinforcement learning problem becomes impossible to solve (Armstrong & Mindermann, 2018).”
Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “Research in imitation and inverse reinforcement learning, with robustness to noise is an emerging area (Evans, 2016; Nair et al., 2017), but require expert data.”
Machine Theory of Mind (2018, methodology): “, 2000; Abbeel & Ng, 2004), Bayesian inference (Lucas et al., 2014; Evans et al., 2016), Bayesian Theory of Mind (Baker et al.”
Inverse Reward Design (2017, background): “In inverse reinforcement learning (IRL) (Ng & Russell, 2000; Ziebart et al., 2008; Evans et al., 2016; Syed & Schapire, 2007) the agent observes demonstrations of (approximately) optimal behavior and infers the reward function being optimized.”
Occam's razor is insufficient to infer the preferences of irrational agents (2017, methodology): “Evans et al. [2015b] and Evans et al. [2015a] use Bayesian inference to identify speciﬁc planning biases such as myopic planning and hyperbolic time-discounting.”
Active Reinforcement Learning: Observing Rewards at a Cost (2020, methodology): “The agent’s objective is to maximize total reward minus total query cost.”
Active Reinforcement Learning with Monte-Carlo Tree Search (2018, background): “In Inverse Reinforcement Learning (IRL) and Imitation Learning (Abbeel and Ng, 2004; Ho and Ermon, 2016; Evans et al., 2016), humans provide demonstrations that ar X iv :1 80 3.”
AGI Safety Literature Review (2018, methodology): “To address this, Evans et al. (2016) develop a method for learning the reward function of agents exhibiting some human-like irrationalities.”
Alignment for Advanced Machine Learning Systems (2020, background): “…due to their reliance on the faulty assumption that human demonstrators are optimizing for a reward function, where in reality humans are often irrational, ill-informed, incompetent, and immoral; recent work by Evans, Stuhlm¨uller, and Goodman (2015b, 2015a) has begun to address these issues.”
FHI Oxford Technical Report # 2018-2 Predicting Human Deliberative Judgments with Machine Learning (2018, background): “This approach of imitating human decision-making is only promising if it imitates human deliberate (slow) judgments [24, 25].”