One-Shot Imitation Learning

By Yan Duan, Marcin Andrychowicz, Bradly C. Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, Wojciech Zaremba. 2017

745 citations recorded by the source corpus.

Imitation learning has been commonly applied to solve different tasks in isolation. This usually requires either careful feature engineering, or a significant number of samples. This is far from what we desire: ideally, robots should be able to learn from very few demonstrations of any given task, and instantly generalize to new situations of the same task, without requiring task-specific engineering. In this paper, we propose a meta-learning framework for achieving such capability, which we call one-shot imitation learning. Specifically, we consider the setting where there is a very large set of tasks, and each task has many instantiations. For example, a task could be to stack all blocks on a table into a single tower, another task could be to place all blocks on a table into two-block towers, etc. In each case, different instances of the task would consist of different sets of blocks with different initial states. At training time, our algorithm is presented with pairs of demonstrations for a subset of all tasks. A neural net is trained that takes as input one demonstration and the current state (which initially is the initial state of the other demonstration of the pair), and outputs an action with the goal that the resulting sequence of states and actions matches as closely as possible with the second demonstration. At test time, a demonstration of a single instance of a new task is presented, and the neural net is expected to perform well on new instances of this new task. The use of soft attention allows the model to generalize to conditions and tasks unseen in the training data. We anticipate that by training this model on a much greater variety of tasks and settings, we will obtain a general system that can turn any demonstrations into robust policies that can accomplish an overwhelming variety of tasks. Videos available at this https URL .

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Graph Attention Networks (2017, methodology): “Similarly, our proposed attention model can be connected to the works by Duan et al. (2017) and Denil et al. (2017), which use a neighborhood attention operation to compute attention coefﬁcients between different objects in an environment.”
Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions (2017, background): “Duan et al. (2017) learn a policy for robotic block stacking conditioned on a small number of demonstration trajectories.”
Relation Networks for Object Detection (2017, neutral): “It follows the recent approaches [13, 49].”
Time-Contrastive Networks: Self-Supervised Learning from Video (2017, methodology): “used for learning robotic skills from expert demonstrations [4, 5, 6, 7] and can be split into two areas: be-”
Adaptive Gradient-Based Meta-Learning Methods (2019, methodology): “GBML is used in a variety of LTL domains such as vision [38, 44, 35], federated learning [16], and robotics [20, 1].”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, background): “The proposed interaction function Υ R-NEM can be seen as a type of Message Passing Neural Network (MPNN; Gilmer et al. (2017)) that incorporates a variant of neighborhood attention (Duan et al., 2017).”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “Duan et al. (2017) present the imitation agent with pairs of demonstrations for each iteration during training, in which the network takes as input the ﬁrsth demonstration and a state sampled from the second demonstration.”
Deep Q-learning From Demonstrations (2017, background): “In one-shot imitation learning (Duan et al. 2017), the agent is provided with an entire demonstration as input in addition to the current state.”
Attention-based Graph Neural Network for Semi-supervised Learning (2018, background): “Here we note that independently from this work attention over sets has been proposed as “neighborhood attention” [16, 25] for a different application.”
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020, methodology): “The task can be represented as a full video [41], a sequence of image keyframes [16], or a trajectory in state space [10], while the training objective can”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “A number of works have studied learning representations and policies that transfer across domains, including transfer from simulation to the real world [1, 33, 34], transfer across different dynamics [35, 36, 37, 2], transfer across robot morphologies with invariant feature spaces [38] and modularit”
The MAGICAL Benchmark for Robust Imitation (2020, background): “Meta-IL [15, 25] and meta-IRL [38, 39] algorithms assume that a large body of demonstrations is available for some set of “train tasks”, but only a few demonstrations are available for “test tasks” that might be encountered in the future.”
One-Shot Visual Imitation Learning via Meta-Learning (2017, methodology): “A common approach, often referred to as contextual policies, is to provide the task as an input to the policy or value function, where the task is represented as a goal or demonstration [1, 2, 4, 3, 5].”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, methodology): “Unlike prior work in fewshot imitation learning [6, 9, 43, 17, 27], our method enables the agent to additionally improve upon trial-and-error experience.”
Overcoming Exploration in Reinforcement Learning with Demonstrations (2017, background): “One-shot Imitation [4] learns to stack blocks in a way that generalizes to new target configurations, but uses more than 100,000 demonstrations to train the system.”
Emergent Tool Use From Multi-Agent Autocurricula (2019, methodology): “We utilize entity-centric observations [70, 71] and use attention mechanisms to capture object-level information [72, 73].”
Machine Theory of Mind (2018, methodology): “The observer must learn to predict the behaviour of many agents, whose rewards, parameterisations, and policies may vary considerably; in this respect, the problem resembles the one-shot imitation learning setup recently introduced in Duan et al. (2017) and Wang et al. (2017).”
Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, methodology): “When expert actions or expert policies are available, behavioral cloning can be used (Rahmatizadeh et al. [36], James et al. [17], Duan et al. [5]).”