Adversarial Imitation via Variational Inverse Reinforcement Learning

By Ahmed Hussain Qureshi, Michael C. Yip. 2018

65 citations recorded by the source corpus.

We consider a problem of learning the reward and policy from expert examples under unknown dynamics. Our proposed method builds on the framework of generative adversarial networks and introduces the empowerment-regularized maximum-entropy inverse reinforcement learning to learn near-optimal rewards and policies. Empowerment-based regularization prevents the policy from overfitting to expert demonstrations, which advantageously leads to more generalized behaviors that result in learning near-optimal rewards. Our method simultaneously learns empowerment through variational information maximization along with the reward and policy under the adversarial learning formulation. We evaluate our approach on various high-dimensional complex control tasks. We also test our learned rewards in challenging transfer learning problems where training and testing environments are made to be different from each other in terms of dynamics or structure. The results show that our proposed method not only learns near-optimal rewards and policies that are matching expert behavior but also performs significantly better than state-of-the-art inverse reinforcement learning algorithms.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Statistical and Nonlinear Physics

Citations in the corpus

The MAGICAL Benchmark for Robust Imitation (2020, methodology): “As noted in Section 1, several past papers employ “test” variants of standard Gym MuJoCo environments to evaluate IRL generalisation [17, 39, 32, 33], but these modified environments tend to have trivial reward functions (e.”
Imitation Learning as f-Divergence Minimization (2019, methodology): “AIRL [34], EAIRL [35] uses this to recover rewards.”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, neutral): “Most deep learning-based methods for reward learning require access to demonstrator actions and do not scale to high-dimensional tasks such as video games (e.g. Atari) (Ho & Ermon, 2016; Finn et al., 2016; Fu et al., 2017; Qureshi & Yip, 2018).”
Imitation Learning from Video by Leveraging Proprioception (2019, methodology): “, 2019] and improving reward representation [Fu et al., 2018; Qureshi et al., 2019].”
Efficiently Guiding Imitation Learning Algorithms with Human Gaze (2020, methodology): “, 2018] or do not scale to high-dimensional tasks such as video games [Finn et al., 2016; Fu et al., 2017; Qureshi et al., 2018].”