Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation

By Abhishek Gupta, Pieter Abbeel, Sergey Levine. 2017

420 citations recorded by the source corpus.

Imitation learning is an effective approach for autonomous systems to acquire control policies when an explicit reward function is unavailable, using supervision provided as demonstrations from an expert, typically a human operator. However, standard imitation learning methods assume that the agent receives examples of observation-action tuples that could be provided, for instance, to a supervised learning algorithm. This stands in contrast to how humans and animals imitate: we observe another person performing some behavior and then figure out which actions will realize that behavior, compensating for changes in viewpoint, surroundings, object positions and types, and other factors. We term this kind of imitation learning “imitation-from-observation,” and propose an imitation learning method based on video prediction with context translation and deep reinforcement learning. This lifts the assumption in imitation learning that the demonstration should consist of observations in the same environment configuration, and enables a variety of interesting applications, including learning robotic skills that involve tool use simply by observing videos of human tool use. Our experimental results show the effectiveness of our approach in learning a wide range of real-world robotic tasks modeled after common household chores from videos of a human demonstrator, including sweeping, ladling almonds, pushing objects as well as a number of tasks in simulation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, background): “One approach developed to address this issue learns a context translation model to translate an observation to predict the observation in the target context [Liu et al., 2018].”
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation (2017, background): “Another approach is to collect third-person demonstrations, such as raw videos [38], [39], but this poses challenges in learning.”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, background): “[35] also learn to extract information from trajectories, but in the context of imitation learning.”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “dition, although we focused on the trajectory-based approach, recent work such as [Finn et al., 2017b, Nair et al., 2017, Liu et al., 2017, Rahmatizadeh et al., 2017] addressed the problem of generalizing skills based on visual information by using a deep learning approach, which is a promising way…”
Playing hard exploration games by watching YouTube (2018, methodology): “In the simple scenario of demonstrations that are aligned frame-by-frame [24, 34], methods such as CCA [2], DCTW [39] or time-contrastive networks (TCN) [34] can be used to learn a common representation space.”
One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning (2018, background): “a reward function based on human demonstrations [47, 49, 26, 46, 50].”
Behavioral Cloning from Observation (2018, methodology): “, 2015] has shown promising performance in simulation [Liu et al., 2017], we generated our demonstrations using agents trained with this method.”
Zero-Shot Visual Imitation (2018, methodology): “Extension to third-person demonstrations (Liu et al., 2018; Stadie et al., 2017) would make the method applicable in more general scenarios.”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, methodology): “Sermanet et al. (2018) and Liu et al. (2018) propose methods to learn directly from a large corpus of videos containing multiple view points of the same task.”
Imitation Learning from Video by Leveraging Proprioception (2019, background): “Liu et al. [ 2018 ] also propose a new architecture to learn a state representation—speciﬁcally, one that is capable of handling viewpoint differences.”
Learning Actionable Representations with Goal-Conditioned Policies (2018, methodology): “Alternative representation learning methods have also been used in the context of imitation learning (Liu et al., 2018) or transfer learning (Gupta et al., 2017).”
PERCEPTUAL GOAL SPECIFICATIONS FOR REINFORCEMENT LEARNING (2017, background): “Many recent works have trained agents such as simulated and real robots from the raw pixels of videos of humans [13, 37, 38, 48].”
Adversarial Imitation Learning from State-only Demonstrations (2019, positive): “This limitation has recently motivated work in the area of imitation from observation (IfO) [8], in which agents seek to perform imitation learning using state-only demonstrations.”
Hybrid Reinforcement Learning with Expert State Sequences (2019, background): “Behavioral Cloning (BC) uses supervised learning, where the learner directly regresses onto the pol-a icy of the expert (Pomerleau 1989; Ross, Gordon, and Bag-nell 2011; Liu et al. 2017).”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, positive): “These limitations have sparked interest IfO [86], [87], where the expert’s actions remain unknown.”
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets (2021, background): “While most prior work on deep visuomotor learning trains a single task in a single domain [7], [8], [9], [10], [11], [12], [13], [14], [15], our goal is not to develop better learning methods, but rather to illustrate how generic multi-domain, multi-task datasets can be used with existing…”
HumanPlus: Humanoid Shadowing and Imitation from Humans (2024, methodology): “Human data has been used extensively for robot learning, including for pre-training visual or intermediate representations or tasks [8, 28, 50, 53, 66, 79, 86, 99] leveraging Internet-scale data [18, 26, 27], and for imitation learning on in-domain human data [3, 12, 43, 47”