Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

By Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Ken Goldberg, Pieter Abbeel. 2017

778 citations recorded by the source corpus.

Imitation learning is a powerful paradigm for robot skill acquisition. However, obtaining demonstrations suitable for learning a policy that maps from raw pixels to actions can be challenging. In this paper we describe how consumer-grade Virtual Reality headsets and hand tracking hardware can be used to naturally teleoperate robots to perform complex tasks. We also describe how imitation learning can learn deep neural network policies (mapping from pixels to actions) that can acquire the demonstrated skills. Our experiments showcase the effectiveness of our approach for learning visuomotor skills.

Subfields: Artificial Intelligence, Control and Systems Engineering, Mechanical Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Mechanical Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, methodology): “There are several approaches to ofﬂine imitation learning [31–34, 9, 10, 25, 35, 36] and ofﬂine reinforcement learning [26, 27, 37–43] (see Appendix A for more discussion on related work).”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “…with only pixel inputs (Mnih et al., 2015, 2016), to learning policies capable of driving real robotic systems in navigation (Zhu et al., 2017b; Zhang et al., 2017a; Tai et al., 2017) and manipulation (Levine et al., 2016; Yu et al., 2018) tasks, the learning power of deep networks drives the…”
Deep Imitative Models for Flexible Inference, Planning, and Control (2018, background): “Imitation learning algorithms use expert demonstration data and, despite similar drift shortcomings (Ross et al., 2011), can sometimes learn effective policies without additional online data collection (Zhang et al., 2018).”
Implicit Behavioral Cloning (2021, methodology): “, compounding errors [2, 3]), in practice it enables some of the most compelling results of real robots generalizing complex behaviors to new unstructured scenarios [4, 5, 6].”
Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control (2021, background): “Recent works [7, 19, 42] use deep neural networks to map directly from image observations to action, and have proven effective for learning visualmotor skills for complex and diverse manipulation tasks.”
Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, background): “Imitation learning (IL) is a powerful paradigm to teach robots to perform manipulation tasks by allowing them to learn from expert demonstrations [32], but IL has mostly been limited to single-arm manipulation tasks [26, 41].”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, methodology): “A policy can be learned without any interaction, as in Behavioral Cloning (BC) [27, 33, 41] or using additional interaction with the environment, as in Inverse Reinforcement Learning (IRL) [1, 2].”
Visual Imitation Made Easy (2020, background): “We care more about the direction between the prediction and ground truth actions than the magnitude, so we add the following loss to encourage this directional alignment [5].”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, methodology): “The model is close to [1]–[3] with the key architectural traits of having a few convolutional layers followed by a channel-wise spatial expectation operation, which has been widely used [10], [11], [25], [29], [36]–[38].”
Simitate: A Hybrid Imitation Learning Benchmark (2019, neutral): “Virtual reality environments have previously been used [40], [41] for evaluation of human robot interfaces.”
ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation (2018, methodology): “Interestingly, Zhang et al. [34] showed that simpler algorithms such as variants of behavior cloning can be made to learn short-horizon manipulation tasks with data on the order of a few hundred VR demonstrations.”
Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation (2018, neutral): “Scaling expert demonstrations has been receiving interest recently with Zhang et al. [38] presenting an approach that using tele-operation to collect demonstrations.”
One-Shot Visual Imitation Learning via Meta-Learning (2017, methodology): “Demonstrations were collected using human teleoperation through a motion controller and virtual reality headset [34], and each demo included the camera video, the sequence of end-effector poses, and the sequence of actions – the end-effector linear and angular velocities.”
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset (2018, background): “Egocentric data has also recently been proven valuable for human-to-robot imitation learning [17], [18], and has a direct impact on HCI applications.”
Learning Latent Plans from Play (2019, methodology): “ral observations, and demonstrations must be provided manually for each task. When using high-capacity models, hundreds or even thousands of demonstrations may be required for each task (Zhang et al. [66], Rahmatizadeh et al. [45], Rajeswaran et al. [47], Duan et al. [11]). In this paper, we instead”
One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning (2018, methodology): “mount of data per-object is quite low (around 10 trials), which is one or two orders of magnitude less than the number of demonstrations per-object used in recent single-task imitation learning works [36, 62]. Thus, if the goal is to enable a generalist robot that can adapt to a diverse range of obj”
Task-Embedded Control Networks for Few-Shot Imitation Learning (2018, background): “However, these solutions often learn their skills from scratch and need a large amount of training data [3, 4, 5].”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “When combined with deep function approximators, IL typically requires many human demonstrations [38, 37, 53] to drive supervised learning of a policy.”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “Deep learning has enabled imitation learning directly 53 from raw image observations [8, 16, 17].”