Zero-Shot Visual Imitation

By Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A. Efros, Trevor Darrell. 2018

329 citations recorded by the source corpus.

Imitating expert demonstration is a powerful mechanism for learning to perform tasks from raw sensory observations. The current dominant paradigm in learning from demonstration (LfD) [3,16,19,20] requires the expert to either manually move the robot joints (i.e., kinesthetic teaching) or teleoperate the robot to execute the desired task. The expert typically provides multiple demonstrations of a task at training time, and this generates data in the form of observation-action pairs from the agent's point of view. The agent then distills this data into a policy for performing the task of interest. Such a heavily supervised approach, where it is necessary to provide demonstrations by controlling the robot, is incredibly tedious for the human expert. Moreover, for every new task that the robot needs to execute, the expert is required to provide a new set of demonstrations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Learning to Understand Goal Specifications by Modelling Reward (2018, background): “Several recent imitation learning works consider using goal-states directly for deﬁning the task (Ganin et al., 2018; Pathak et al., 2018).”
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020, methodology): “be either maximizing the likelihood of action given an expert trajectory as in [10, 41, 16, 30], matching the distribution of demonstrations (GAIL [40]), or following a trajectory based on learned dynamics models (AVID [38]).”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “Further, we also study how we can extend vision-based inverse models [10, 27, 6, 11] for generalizable robot-agnostic control.”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “Pathak et al. (2018) uses an auto-regressive recurrent model to predict a multi-step loss on a trajectory, which is closely related to ours.”
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, background): “Recently, the success of deep learning has garnered increased interest in learning to solve perception and manipulation tasks in an end-to-end framework [11, 12, 28, 29, 30].”
Visual Reinforcement Learning with Imagined Goals (2018, background): “While prior works on vision-based deep reinforcement learning for robotics can efficiently learn a variety of behaviors such as grasping [33, 32, 24], pushing [1, 8, 10], navigation [30, 21], and other manipulation tasks [26, 23, 30], they each make assumptions that limit their applicability to trai”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “Note that differently from some robotic navigation settings [36], here the agent is not provided with a goal image, but must identify from the textual description and environment whether it has reached the goal.”
Learning Latent Plans from Play (2019, background): “Specifically, it is not to be confused with recent “self-supervised robotics” work, where unattended, scripted robotic data collection is used to learn skills such as grasping ([1, 35, 42, 31, 13, 39]) failure recovery.”
Playing hard exploration games by watching YouTube (2018, background): “Other approaches include interacting with the environment before introducing the expert demonstrations [38, 31] and goal conditioned policies for high ﬁdelity imitation [29], although these papers typically do not assume domain gap or operate in sparse reward settings.”
Curiosity Driven Exploration of Learned Disentangled Goal Spaces (2018, neutral): “This has also been tackled under the terminology of multi-modality in [29].”
Large-Scale Study of Curiosity-Driven Learning (2018, background): “Indeed, there is evidence that pre-training an agent on a given environment using only intrinsic rewards allows it to learn much faster when ﬁne-tuned to a novel task in a novel environment [27, 28].”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “Multiple prior imitation learning works have achieved different forms of generalization, including one-shot generalization to novel objects [1, 2, 3, 4, 18], to novel object configurations [19], and to novel goal configurations [6, 7, 20], as well as zero-shot generalization to new objects [5], scen”
Hindsight policy gradients (2017, background): “Learning such goal-conditional behavior has received significant attention in machine learning and robotics, especially because a goal-conditional policy may generalize desirable behavior to goals that were never encountered by the agent (Schmidhuber & Huber, 1990; Da Silva et al., 2012; Kupcsik et ”
One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL (2018, methodology): “Inverse dynamics models: Our method is closely related to recent work on learned inverse dynamics models (Pathak et al., 2018; Nair et al., 2017a).”
Learning to Reach Goals via Iterated Supervised Learning (2019, methodology): “We additionally consider limited-horizon relabeling, in which only states and goals that are at most 3 steps apart are relabeled, similar to proposals in prior work [Pathak et al., 2018, Savinov et al., 2018].”
Learning To Reach Goals Without Reinforcement Learning (2019, methodology): “We see that this performs poorly as compared to GCSL, indicating that multi-horizon relabeling is an important component for effective learning in comparison to one-step inverse models (Pathak et al., 2018).”
Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data (2020, methodology): “We primarily focus on learning-based methods, which have been shown to generalize to a variety of tasks [28, 30, 41].”
Learning to Follow Language Instructions with Adversarial Reward Induction (2018, background): “Several recent imitation learning works considered using goal-states directly for defining the task [10, 21].”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, background): “Pathak et al. 50 and Nair et al. 51 train goal conditioned policies to take actions that move towards expert-provided goal states expressed as high dimensional visual waypoints.”