Cooperative Inverse Reinforcement Learning

By Dylan Hadfield-Menell, Stuart J. Russell, Pieter Abbeel, Anca Dragan. 2016

776 citations recorded by the source corpus.

For an autonomous system to be helpful to humans and to pose no unwarranted risks, it needs to align its values with those of the humans in its environment in such a way that its actions contribute to the maximization of value for the humans. We propose a formal definition of the value alignment problem as cooperative inverse reinforcement learning (CIRL). A CIRL problem is a cooperative, partial-information game with two agents, human and robot; both are rewarded according to the human's reward function, but the robot does not initially know what this is. In contrast to classical IRL, where the human is assumed to act optimally in isolation, optimal CIRL solutions produce behaviors such as active teaching, active learning, and communicative actions that are more effective in achieving value alignment. We show that computing optimal joint policies in CIRL games can be reduced to solving a POMDP, prove that optimality in isolation is suboptimal in CIRL, and derive an approximate CIRL algorithm.

Subfields: Artificial Intelligence, Management Information Systems, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Information Systems

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, background): “Our work could also be seen of a specific instance of the cooperative inverse reinforcement learning framework (Hadfield-Menell et al., 2016).”
Alignment of Language Agents (2021, background): “Another class of approaches (Hadfield-Menell et al., 2016, 2017b) contains an agent which is uncertain about its objective, and aims for the agent to correctly calibrate its beliefs about it, and in doing so avoid gaming it.”
Scalable agent alignment via reward modeling: a research direction (2018, methodology): “Motivated by this second shortcoming of IRL, Hadﬁeld-Menell et al. (2016) propose cooperative inverse reinforcement learning (CIRL).”
Supervising strong learners by amplifying weak experts (2018, background): “Inverse reinforcement learning: by observing human behavior and inferring the underlying reward function that they are optimizing,(Ng et al., 2000; Hadﬁeld-Menell et al., 2016) inverse reinforcement learning could also potentially learn reward functions for tasks that are too challenging for humans.”
Improving alignment of dialogue agents via targeted human judgements (2022, background): “However, dialogue for supervision also needs cooperation between humans and agents to jointly clarify what is meant, and avoid misunderstandings or gaps (Hadfield-Menell et al., 2016; Russell, 2020).”
Concrete Problems in AI Safety (2016, background): “For example, the agent could learn to identify “salient” events in the environment, and request to see the reward associated with these events.”
Unsolved Problems in ML Safety (2021, background): “Other directions include learning difficult-to-specify goals in interactive environments [70], learning the idiosyncratic values of different stakeholders [121], and learning about cosmopolitan goals such as endowing humans with the capabilities necessary for high welfare [133].”
Handling Model Uncertainty and Multiplicity in Explanations via Model Reconciliation (2018, background): “There has been some recent work (Nikolaidis et al. 2015; Hadﬁeld-Menell et al. 2016) that aims to learn human mental models iteratively during the course of interactions especially when there is uncertainty about human preferences.”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, neutral): “required. Noise injection during expert demonstrations has also been proposed in order to correct for covariate shift [22]. Other paradigms for human-in-the-loop policy learning include collaboration [14,15], teaching the robot through informative sample selection [7,9,17], and leveraging physical k”
The MAGICAL Benchmark for Robust Imitation (2020, background): “However, it’s worth noting that unlike, say, HAMDPs [16] or CIRL games [21], IL cannot request clarification from a demonstrator if the supplied demonstrations are ambiguous, which limits its ability to learn the right objective in general.”
Machine Theory of Mind (2018, background): “Such models will be important for value alignment (Hadﬁeld-Menell et al., 2016) and ﬂexible cooperation (Nowak, 2006; Kleiman-Weiner et al., 2016; Barrett et al., 2017; Kris Cao), and will likely be an ingredient in future machines’ ethical decision making (Churchland, 1996).”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “Hadfield-Menell et al. [2016] show that finding optimal policies for the human and robot in CIRL corresponds to solving a POMDP. A drawback of the POMDP model is that in practice exact optimal solutions for the model are hard to come by but the POMDP model can be used as a theoretical tool and a bas”
Inverse Reward Design (2017, methodology): “Ultimately, we consider both IRD and IRL to be complementary strategies for value alignment (Hadﬁeld-Menell et al., 2016): approaches that allow designers or users to communicate preferences or goals.”
AI Safety Gridworlds (2017, background): “However, looking beyond pure RL agents, Everitt et al. (2017) argue that many RL-inspired frameworks such as cooperative inverse RL (Hadfield-Menell et al., 2016a), learning from human preferences (see e.g. Akrour et al., 2012, Wilson et al., 2012 and Christiano et al., 2017), and learning values…”
Reinforcement Learning with a Corrupted Reward Channel (2017, methodology): “The more recently proposed (cooperative) IRL framework may be seen as an approach to build CR agents [Hadfield-Menell et al., 2016, 2017].”
Consequences of Misaligned AI (2021, background): “behavior for specified objectives, without corresponding advancements in our ability to avoid or correct specification errors, will amplify the harms of AI systems (8; 19).”
An Introduction to Deep Reinforcement Learning (2018, neutral): “In the work of Hadfield-Menell et al., 2016, the cooperative inverse reinforcement learning framework considers a two-player game between a human and a robot interacting with an environment with the purpose”