Robust Imitation of Diverse Behaviors

By Ziyun Wang, Josh Merel, Scott E. Reed, Nando de Freitas, Greg Wayne, Nicolas Manfred Otto Heess. 2017

227 citations recorded by the source corpus.

Deep generative models have recently shown great promise in imitation learning for motor control. Given enough data, even supervised approaches can do one-shot imitation learning; however, they are vulnerable to cascading failures when the agent trajectory diverges from the demonstrations. Compared to purely supervised methods, Generative Adversarial Imitation Learning (GAIL) can learn more robust controllers from fewer demonstrations, but is inherently mode-seeking and more difficult to train. In this paper, we show how to combine the favourable aspects of these two approaches. The base of our model is a new type of variational autoencoder on demonstration trajectories that learns semantic policy embeddings. We show that these embeddings can be learned on a 9 DoF Jaco robot arm in reaching tasks, and then smoothly interpolated with a resulting smooth interpolation of reaching behavior. Leveraging these policy representations, we develop a new version of GAIL that (1) is much more robust than the purely-supervised controller, especially with few demonstrations, and (2) avoids mode collapse, capturing many diverse behaviors when GAIL on its own does not. We demonstrate our approach on learning diverse gaits from demonstration on a 2D biped and a 62 DoF 3D humanoid in the MuJoCo physics environment.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “Several extensions of GAIL have also been proposed (Baram et al., 2016; Wang et al., 2017).”
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020, methodology): “be either maximizing the likelihood of action given an expert trajectory as in [10, 41, 16, 30], matching the distribution of demonstrations (GAIL [40]), or following a trajectory based on learned dynamics models (AVID [38]).”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “As a second baseline and to test the quality of OPAL-extracted primitives, we experiment with an alternative unsupervised objective from Wang et al. (2017), which prescribes using a sequential VAE (SVAE) over state trajectories in conjunction with imitation learning.”
Reinforcement Learning from Imperfect Demonstrations (2018, background): “Recently, (Ho & Ermon, 2016; Wang et al., 2017; Ziebart et al., 2008) explore an adversarial paradigm for the behavior cloning method.”
Machine Theory of Mind (2018, methodology): “This problem resembles one-shot imitation learning (Duan et al., 2017; Wang et al., 2017), but differs since the objective is not for the observer to execute the behaviours itself.”
Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, background): “Imitation can effectively shape the agent’s behaviors towards the demonstration trajectories [51].”
Learning Latent Plans from Play (2019, neutral): “In contrast to prior work on few-shot learning from demonstration ([29, 30]), our method does not require a meta-training phase, any expensive task-speciﬁc demonstrations, or a predeﬁned task distribution.”
Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning (2018, background): “More recent work extends the framework by improving on stability and robustness (Wang et al., 2017; Kim & Park, 2018) and making connections to model-based imitation learning (Baram et al., 2017).”
Neural probabilistic motor primitives for humanoid control (2018, background): “More similar to our work is the setting examined by Wang et al. (2017), in which full-body humanoid movements were studied.”
Ego-Pose Estimation and Forecasting As Real-Time PD Control (2019, methodology): “Approaches based on GAIL [14] have also been proposed to eliminate the need for manual reward engineering [33, 60, 67].”
Generating Multi-Agent Trajectories using Programmatic Weak Supervision (2018, background): “…approaches to learning interpretable intermediate variables typically focus on learning disentangled latent representations in an unsupervised way (e.g., (Li et al., 2017; Wang et al., 2017)), but it is challenging for such approaches to handle complex sequential settings (Chen et al., 2017).”
One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL (2018, background): “A few recent works have explored one-shot task-based imitation learning (Finn et al., 2017; Duan et al., 2017; Wang et al., 2017), i.”
DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations (2017, background): “There are also several approaches that consider “demonstration clustering”, that is, identifying clusters of already segmented demonstrations [25, 26, 27, 28].”
Meta-learning of Sequential Strategies (2019, methodology): “…the initial network parameters [Finn et al., 2017, Nichol and Schulman, 2018], the learning targets [Xu et al., 2018], conditional distributions [Wang et al., 2017, Garnelo et al., 2018, Gordon et al., 2018, Zintgraf et al., 2018, Chen et al., 2018a], or even the entire learning procedure using…”
Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, background): “A popular approach for fast adaptation to new environments is meta reinforcement learning (meta RL) (Cully et al., 2015; Finn et al., 2017; Wang et al., 2017; Duan et al., 2016; Xu et al., 2018; Houthooft et al., 2018; Sæmundsson et al., 2018; Nagabandi et al., 2018; Humplik et al., 2019; Rakelly et”
Multi-task Maximum Entropy Inverse Reinforcement Learning (2018, methodology): “Wang et al. (2017) use GAIL with the discriminator conditioned on the output generated by an LSTM encoder.”
OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning (2017, methodology): “Several works have investigated learning a latent structure for such a purpose (Merel et al. 2017; Hausman et al. 2017; Wang et al. 2017).”
A Survey of Imitation Learning Methods, Environments and Metrics (2024, background): “Towards sample efficiency and the issue of saddle points, Wang et al. [112] use a variational autoencoder to increase diversity with relatively fewer demonstrations and achieve one-shot learning to new trajectories.”