Third-Person Imitation Learning

By Bradly C. Stadie, Pieter Abbeel, Ilya Sutskever. 2017

246 citations recorded by the source corpus.

Reinforcement learning (RL) makes it possible to train agents capable of achieving sophisticated goals in complex and uncertain environments. A key difficulty in reinforcement learning is specifying a reward function for the agent to optimize. Traditionally, imitation learning in RL has been used to overcome this problem. Unfortunately, hitherto imitation learning methods tend to require that demonstrations are supplied in the first-person: the agent is provided with a sequence of states and a specification of the actions that it should have taken. While powerful, this kind of imitation learning is limited by the relatively hard problem of collecting first-person demonstrations. Humans address this problem by learning from third-person demonstrations: they observe other humans perform tasks, infer the task, and accomplish the same task themselves. In this paper, we present a method for unsupervised third-person imitation learning. Here third-person refers to training an agent to correctly achieve a simple goal in a simple environment when it is provided a demonstration of a teacher achieving the same goal but from a different viewpoint; and unsupervised refers to the fact that the agent receives only these third-person demonstrations, and is not provided a correspondence between teacher states and student states. Our methods primary insight is that recent advances from domain confusion can be utilized to yield domain agnostic features which are crucial during the training process. To validate our approach, we report successful experiments on learning from third-person demonstrations in a pointmass domain, a reacher domain, and inverted pendulum.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, background): “, 2016), imitation learning (Ho and Ermon, 2016; Stadie et al., 2017), semi-supervised skill generalization (Finn et al.”
One-Shot Imitation Learning (2017, background): “Success stories include domain adaptation in computer vision [64, 34, 28, 4, 15, 24, 33, 59, 14] and control [60, 45, 46, 20, 54].”
Scalable agent alignment via reward modeling: a research direction (2018, methodology): “There are at least two approaches to leverage this existing data: using unsupervised learning (such as unsupervised pretraining or third-person imitation learning; Stadie et al., 2017) or by manually annotating it.”
Learning to Understand Goal Specifications by Modelling Reward (2018, background): “Future work could focus on applying third-person imitation learning methods recently proposed by Stadie et al. (2017) learn the aforementioned invariances.”
Imitating Interactive Intelligence (2020, background): “Further, it is at present intrinsically difficult to do so because algorithms and systems have not yet been developed that can perceive and understand the intentions of humans in a way that transfers across radical changes in embodiment, environment, and perspective (Stadie et al., 2017; Borsa et al”
Time-Contrastive Networks: Self-Supervised Learning from Video (2017, background): “from a different viewpoint or an agent with a different embodiment, such as a human [13, 14, 15].”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “For manipulation, Stadie et al. (2017) extend the GAIL formulation with ideas from domain confusion loss (Tzeng et al., 2014), and successfully utilize it to train agents to imitate third-person demonstrations, by learning a domain-agnostic representation of the agent’s observations.”
Visual Imitation Made Easy (2020, background): “There have also been recent efforts to imitate from visual demonstrations collected from a different space e.g. from a different viewpoint or an agent with a different embodiment [6, 11] from the robot.”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, background): “Consequently, the extracted features would be invariant with respect to the viewpoint [Stadie et al., 2017].”
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation (2017, background): “Another approach is to collect third-person demonstrations, such as raw videos [38], [39], but this poses challenges in learning.”
An Algorithmic Perspective on Imitation Learning (2018, background): “Recent work on third-person imitation learning [Stadie et al., 2017] addresses this problem in some simple environments.”
Playing hard exploration games by watching YouTube (2018, methodology): “We also experimented with third-person imitation methods [37] that combine the ideas of generative adversarial imitation learning (GAIL) [21] and adversarial domain confusion [16, 40], but were unable to make progress using the very long YouTube demonstration trajectories.”
One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning (2018, background): “a reward function based on human demonstrations [47, 49, 26, 46, 50].”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “Many DRL papers in continuous control [103, 37, 67, 72, 4, 111] have used the MuJoCo physics engine to obtain relatively realistic dynamics for multi-joint continuous control problems [122], and there has now been some effort”
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task (2017, methodology): “ ability to explore sufﬁciently can become intractable as the complexity of the task increases. To counter this, imitation learning can be used by providing demonstrations in ﬁrst [16,17,18] or third [19] person perspective. Our method uses the full state of the simulation to effectively produce ﬁrs”
Robust Imitation of Diverse Behaviors (2017, background): “GAIL has become a popular choice for imitation learning [16] and there already exist model-based [3] and third-person [35] extensions.”
Zero-Shot Visual Imitation (2018, methodology): “Extension to third-person demonstrations (Liu et al., 2018; Stadie et al., 2017) would make the method applicable in more general scenarios.”
Task-Relevant Adversarial Imitation Learning (2019, methodology): “Following the notation of Sutton and Barto [2018], a Markov Decision Process (MDP) is a tuple (S,A, R, P, γ) with states S, actions A, reward function R(s, a), transition distribution P (s′|s, a), and discount γ.”
Imitating Latent Policies from Observation (2018, background): “Finally, both third-person imitation learning (Stadie et al., 2017) and GAIfO (Torabi et al.”