Occam's razor is insufficient to infer the preferences of irrational agents

By Stuart Armstrong, Sören Mindermann. 2017

97 citations recorded by the source corpus.

Inverse reinforcement learning (IRL) attempts to infer human rewards or preferences from observed behavior. Since human planning systematically deviates from rationality, several approaches have been tried to account for specific human shortcomings. However, the general problem of inferring the reward function of an agent of unknown rationality has received little attention. Unlike the well-known ambiguity problems in IRL, this one is practically relevant but cannot be resolved by observing the agent's policy in enough environments. This paper shows (1) that a No Free Lunch result implies it is impossible to uniquely decompose a policy into a planning algorithm and reward function, and (2) that even with a reasonable simplicity prior/Occam's razor on the set of decompositions, we cannot distinguish between the true decomposition and others that lead to high regret. To address this, we need simple `normative' assumptions, which cannot be deduced exclusively from observations.

Subfields: Cognitive Neuroscience, General Decision Sciences, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In General Decision Sciences

In Alignment & RLHF

Citations in the corpus

Artificial Intelligence, Values, and Alignment (2020, background): “This means that it is very hard to make reliable inferences from observed behaviour, and it may be impossible to do so without making controversial assumptions about human rationality (Sen, 1973; Armstrong and Mindermann, 2018).”
Scalable agent alignment via reward modeling: a research direction (2018, background): “Even some assumptions about the human’s rationality can be relaxed (Evans et al., 2016), but in full generality the inverse reinforcement learning problem becomes impossible to solve (Armstrong & Mindermann, 2018).”
Scaling Laws for Reward Model Overoptimization (2022, background): “This contains issues ranging from the mundane, such as labellers choosing options that only appear to match their intent 14 , to substantially more philosophically fraught issues [Armstrong and Mindermann, 2018, Sunstein et al., 2001].”
Towards Understanding Sycophancy in Language Models (2023, unknown): “Moreover, modeling human preferences presents some challenges (Zhao et al., 2016; Hong et al., 2022; Lindner & El-Assady, 2022; Mindermann & Armstrong, 2018; Shah et al., 2019).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “In particular, modeling human irrationalities can make reward learning difficult [Nguyen et al., 2017, Mindermann and Armstrong, 2018, Shah et al., 2019], leading to a trade-off between efficiency and accuracy.”
Theory of Mind and Preference Learning at the Interface of Cognitive Science, Neuroscience, and AI: A Review (2022, background): “On the other hand, some theoretical results (Mindermann and Armstrong, 2018) cast doubt on the possibility of completely inferring human preferences from observed behavior only, even with access to infinite amounts of behavior data.”
Embedded Agency (2019, background): “But the successor agent has to ﬁgure out what to do in situations that the initial agent can’t even understand, and try to respect the goals of something that the successor can see is inconsistent, which also seems very hard (Armstrong and Mindermann 2017).”