Learning a Prior over Intent via Meta-Inverse Reinforcement Learning

By Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, Chelsea Finn. 2018

69 citations recorded by the source corpus.

A significant challenge for the practical application of reinforcement learning in the real world is the need to specify an oracle reward function that correctly defines a task. Inverse reinforcement learning (IRL) seeks to avoid this challenge by instead inferring a reward function from expert behavior. While appealing, it can be impractically expensive to collect datasets of demonstrations that cover the variation common in the real world (e.g. opening any type of door). Thus in practice, IRL must commonly be performed with only a limited set of demonstrations where it can be exceedingly difficult to unambiguously recover a reward function. In this work, we exploit the insight that demonstrations from other tasks can be used to constrain the set of possible reward functions by learning a "prior" that is specifically optimized for the ability to infer expressive reward functions from limited numbers of demonstrations. We demonstrate that our method can efficiently recover rewards from images for novel tasks and provide intuition as to how our approach is analogous to learning a prior.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

The MAGICAL Benchmark for Robust Imitation (2020, background): “[38] and Gleave and Habryka [18] use gridworld benchmarks to evaluate meta- and multi-task IRL, and both benchmarks draw a distinction between demonstration and execution environments within a meta-testing task.”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “In reality, tasks have considerable shared structure, and a classifier that incorporates data from multiple tasks, analogously to prior work on metalearning [42, 43], could in principle further reduce the number of queries and examples that is needed.”
Guiding Policies with Language via Meta-Learning (2018, background): “, 2017), as well as goal specification (Xu et al., 2018; Xie et al., 2018), to our knowledge no prior work has proposed meta-training of policies that can acquire new tasks from iterative language corrections.”
Meta-Inverse Reinforcement Learning with Probabilistic Context Variables (2019, background): “Meta-IRL [36, 12] incorporates meta-learning and IRL, showing fast adaptation of the reward functions to unseen tasks.”
Meta Reinforcement Learning with Task Embedding and Shared Policy (2019, methodology): “, 2018], and the reward functions in the context of unsupervised learning and inverse RL respectively [Gupta et al., 2018a; Xu et al., 2018a].”
What Can Learned Intrinsic Rewards Capture? (2019, background): “, 2019), and inverse RL (Xu et al., 2019).”
Policy Gradient Bayesian Robust Optimization for Imitation Learning (2021, background): “This distribution could be a prior distribution P(R) that the agent learns from previous tasks (Xu et al., 2019).”
Theory of Mind and Preference Learning at the Interface of Cognitive Science, Neuroscience, and AI: A Review (2022, background): “Xu et al. (2019) adapts MAML to maximum entropy IRL (Ziebart et al., 2008) by learning a “prior” over reward functions which is speciﬁcally optimized so that the reward function corresponding to a new task can be learned from a limited number of demonstrations.”
Bayesian Robust Optimization for Imitation Learning (2020, background): “This prior could come from historical data or from meta-learning on similar tasks [60].”