Multi-task Maximum Entropy Inverse Reinforcement Learning

By Adam Gleave, Oliver Habryka. 2018

42 citations recorded by the source corpus.

Multi-task Inverse Reinforcement Learning (IRL) is the problem of inferring multiple reward functions from expert demonstrations. Prior work, built on Bayesian IRL, is unable to scale to complex environments due to computational constraints. This paper contributes a formulation of multi-task IRL in the more computationally efficient Maximum Causal Entropy (MCE) IRL framework. Experiments show our approach can perform one-shot imitation learning in a gridworld environment that single-task IRL algorithms need hundreds of demonstrations to solve. We outline preliminary work using meta-learning to extend our method to the function approximator setting of modern MCE IRL algorithms. Evaluating on multi-task variants of common simulated robotics benchmarks, we discover serious limitations of these IRL algorithms, and conclude with suggestions for further work.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

The MAGICAL Benchmark for Robust Imitation (2020, background): “However, it’s worth noting that unlike, say, HAMDPs [16] or CIRL games [21], IL cannot request clarification from a demonstrator if the supplied demonstrations are ambiguous, which limits its ability to learn the right objective in general.”
Meta-Inverse Reinforcement Learning with Probabilistic Context Variables (2019, background): “Meta-IRL [36, 12] incorporates meta-learning and IRL, showing fast adaptation of the reward functions to unseen tasks.”
Learning to Generalize from Sparse and Underspecified Rewards (2019, background): “…the automatic discovery of goal speciﬁ-cations (Xie et al., 2018; Bahdanau et al., 2019), text generation tasks (Wang et al., 2018; Wu et al., 2017; Bosselut et al., 2018) and the optimization of reward functions ( e.g., Gleave & Habryka (2018); Fu et al. (2019); Shi et al. (2018)) via inverse RL.”
Few-Shot Goal Inference for Visuomotor Learning and Planning (2018, background): “…goals to robots is related to that of prior works on one-shot imitation learning [40, 41, 42] and recent work on meta-inverse reinforcement learning [43, 44], both of which strive to learn behavior from one or a few demonstrations either through direct imitation or by ﬁrst learning a reward.”