AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos

By Laura M. Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, Sergey Levine. 2019

188 citations recorded by the source corpus.

Robotic reinforcement learning (RL) holds the promise of enabling robots to learn complex behaviors through experience. However, realizing this promise for long-horizon tasks in the real world requires mechanisms to reduce human burden in terms of defining the task and scaffolding the learning process. In this paper, we study how these challenges can be alleviated with an automated robotic learning framework, in which multi-stage tasks are defined simply by providing videos of a human demonstrator and then learned autonomously by the robot from raw image observations. A central challenge in imitating human videos is the difference in appearance between the human and robot, which typically requires manual correspondence. We instead take an automated approach and perform pixel-level image translation via CycleGAN to convert the human demonstration into a video of a robot, which can then be used to construct a reward function for a model-based RL algorithm. The robot then learns the task one stage at a time, automatically learning how to reset each stage to retry it multiple times without human-provided resets. This makes the learning process largely automatic, from intuitive task specification via a video to automated training with minimal human intervention. We demonstrate that our approach is capable of learning complex tasks, such as operating a coffee machine, directly from raw image observations, requiring only 20 minutes to provide human demonstrations and about 180 minutes of robot interaction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020, methodology): “be either maximizing the likelihood of action given an expert trajectory as in [10, 41, 16, 30], matching the distribution of demonstrations (GAIL [40]), or following a trajectory based on learned dynamics models (AVID [38]).”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “Positive examples may be designated or reinforced through human feedback (Zhang et al., 2019; Singh et al., 2019; Smith et al., 2020).”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “LOReL [81] crowd-sources humans labels to train language-conditioned reward function for ofﬂine RL. AVID [113] and XIRL [142] extract reward signals via cycle consistency.”
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play (2023, background): “To reduce the domain gap, another thread of work [2, 27, 28, 47, 52, 53, 58] utilizes indomain human videos, where human directly interacts with the robot task environment with their own hands.”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “It has been common to introduce some form of prior derived from human knowledge, which could take the form of corrective interactions [23, 42, 68], structured policy spaces [2, 7, 7, 17, 50, 85, 94, 99, 108, 125], offline robotics data [25, 56, 57, 71, 97], using pretrained visual representations [8”
Transformers for One-Shot Visual Imitation (2020, background): “But how does one represent goal/intention and how can this contextual information be incorporated into the policy function itself? There are three primary approaches in prior work: the first approach is to represent goals/intentions as pixels by generating goal images, and then inferring actions giv”
R3M: A Universal Visual Representation for Robot Manipulation (2022, background): “Several prior works have explored using human video data in robot learning, for example to acquire goals [39, 40, 41], to learn visual dynamics models [42, 43, 44, 45], or to learn representations and rewards [19, 46, 47, 48, 49, 50].”
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (2024, background): “…also focus on multi-task or few-shot imitation learning, [25, 27, 30, 34, 46, 50, 88, 102], language-conditioned imitation learning [12, 47, 82, 83], imitation from play data [21, 57, 74, 89], using human videos [16, 24, 29, 60, 69, 80, 84, 96], and using task-specific structures [49, 83, 103].”
XIRL: Cross-embodiment Inverse Reinforcement Learning (2021, methodology): “An additional category of approaches to third-person imitation learning 95 are those that perform domain adaptation of expert observations [18, 19, 20, 21].”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “Another line of work in imitation learning is to train policies to follow the expert demonstrations [43, 52, 49, 71, 68, 72, 74, 21, 51, 73].”
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators (2023, unknown): “Finally, a promising direction is learning directly from human videos [52], [53], [54], [55], [56], [57].”