Model-based Adversarial Imitation Learning

By Nir Baram, Oron Anschel, Shie Mannor. 2016

46 citations recorded by the source corpus.

Generative adversarial learning is a popular new approach to training generative models which has been proven successful for other related problems as well. The general idea is to maintain an oracle $D$ that discriminates between the expert's data distribution and that of the generative model $G$. The generative model is trained to capture the expert's distribution by maximizing the probability of $D$ misclassifying the data it generates. Overall, the system is \emph{differentiable} end-to-end and is trained using basic backpropagation. This type of learning was successfully applied to the problem of policy imitation in a model-free setup. However, a model-free approach does not allow the system to be differentiable, which requires the use of high-variance gradient estimations. In this paper we introduce the Model based Adversarial Imitation Learning (MAIL) algorithm. A model-based approach for the problem of adversarial imitation learning. We show how to use a forward model to make the system fully differentiable, which enables us to train policies using the (stochastic) gradient of $D$. Moreover, our approach requires relatively few environment interactions, and fewer hyper-parameters to tune. We test our method on the MuJoCo physics simulator and report initial results that surpass the current state-of-the-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “Several extensions of GAIL have also been proposed (Baram et al., 2016; Wang et al., 2017).”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “[4] exploit gradient information that was not used in”
Robust Imitation of Diverse Behaviors (2017, background): “GAIL has become a popular choice for imitation learning [16] and there already exist model-based [3] and third-person [35] extensions.”
Robust motion in-betweening (2020, background): “Imitation-learning based RL approaches [Baram et al. 2016; Ho and Ermon 2016] try to address this with adversarial learning, while others tackle the problem by penalizing distance of a generated state from a reference state [Bergamin et al. 2019; Peng et al. 2018].”