Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets

By Karol Hausman, Yevgen Chebotar, Stefan Schaal, G. Sukhatme, Joseph J. Lim. 2017

164 citations recorded by the source corpus.

Imitation learning has traditionally been applied to learn a single task from demonstrations thereof. The requirement of structured and isolated demonstrations limits the scalability of imitation learning approaches as they are difficult to apply to real-world scenarios, where robots have to be able to execute a multitude of tasks. In this paper, we propose a multi-modal imitation learning framework that is able to segment and imitate skills from unlabelled and unstructured demonstrations by learning skill segmentation and imitation learning jointly. The extensive simulation results indicate that our method can efficiently separate the demonstrations into individual skills and learn to imitate them using a single multi-modal policy. The video of our experiments is available at this http URL

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, neutral): “training in the generative adversarial framework [41, 42, 43].”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “These methods aim to perform imitation learning by learning low-level primitives [9, 26] or latent conditioned policies [23] which meaningfully segment the demonstrations.”
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task (2017, methodology): “ld robots, where the ability to explore sufﬁciently can become intractable as the complexity of the task increases. To counter this, imitation learning can be used by providing demonstrations in ﬁrst [16,17,18] or third [19] person perspective. Our method uses the full state of the simulation to eff”
Robust Imitation of Diverse Behaviors (2017, background): “Two recent GAIL-based approaches [17, 10] introduce additional reward signals that encourage the policy to make use of latent variables which would correspond to different types of demonstrations after training.”
Imitation Learning as f-Divergence Minimization (2019, methodology): “InfoGAN [46] formalizes the GAN framework to recover latent clusters which is then extended to IL [4, 47].”
Imitation Learning via Off-Policy Distribution Matching (2019, neutral): “Many of these applications of the AIL framework (Li et al., 2017; Hausman et al., 2017; Fu et al., 2017) maintain the same form of distribution ratio estimation as GAIL which necessitates on-policy samples.”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “Is it possible instead to learn many skills simultaneously from large amounts of unstructured, unlabeled demonstration data [16, 26, 14]?”
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, background): “[cs.LG] 11 Apr 2019 KEYIN: DISCOVERING SUBGOAL STRUCTURE WITH KEYFRAME-BASED VIDEO PREDICTION once a subgoal sequence is discovered (Konidaris et al., 2012; Niekum et al., 2015; Kroemer et al., 2015; Hausman et al., 2017). Indeed, much of the difﬁculty in learning complex tasks lies in determining w”
Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation (2019, background): “Several multi-modal imitation learning algorithms propose to learn diverse behaviors from demonstrations [25, 26, 27].”
Compositional Imitation Learning: Explaining and executing one task at a time (2018, methodology): “Option discovery has also been addressed in the context of inverse reinforcement learning (IRL) using generative adversarial networks (GANs) [21] to ﬁnd structured policies that are close to demonstration sequences [19].”
Reward Learning from Narrated Demonstrations (2018, background): “Temporal segmentation of sequential demonstrations [23] is easily obtained by considering the segmentation of the transcript into verbal phrases; this alleviates the current need for demonstrations to concern a single isolated task at a time [23].”
Learning Navigation Subroutines from Egocentric Videos (2019, unknown): “Works like [22] extend these formulations to work with trajectory collections that have multiple modes.”
Mutual Alignment Transfer Learning (2017, background): “The approach is conceptually similar to adversarial imitation learning methods [22, 10, 23] which train for one-sided alignment between the agent’s policy and a set of demonstration trajectories.”
OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning (2017, background): “Several works have investigated learning a latent structure for such a purpose (Merel et al. 2017; Hausman et al. 2017; Wang et al. 2017).”
Multi-task Maximum Entropy Inverse Reinforcement Learning (2018, methodology): “Recent extensions to GAIL augment trajectories with a latent intention variable that speciﬁes the task, and then maximise mutual information between the state-action pairs and intention variable (Hausman et al., 2017; Li et al., 2017).”
KeyIn: Discovering Subgoal Structure with Keyframe-based Video Prediction (2019, background): “L G ] 11 A p r 2019 once a subgoal sequence is discovered (Konidaris et al., 2012; Niekum et al., 2015; Kroemer et al., 2015; Hausman et al., 2017).”
Meta-Inverse Reinforcement Learning with Probabilistic Context Variables (2019, background): “Other imitation learning approaches [16, 21, 14, 18] are also based on the adversarial framework, but they do not recover a reward function.”
Imitating Latent Policies from Observation (2018, background): “Other works have aimed to cluster demonstrations, but these approaches have traditionally segmented different types of trajectories, which represent distinct preferences, rather than next-state predictions (Hausman et al., 2017; Babes et al., 2011).”
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale (2021, background): “…and composite skills has been addressed through hierarchical reinforcement learning with options [75, 7, 14], combining multiple sub-tasks [6, 16, 49, 26, 69, 89], reusing samples between tasks [39], relabeling experience in hindsight [3], introducing demonstrations [58, 81, 29, 41, 70, 66, 67].”