One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning

By Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, Sergey Levine. 2018

385 citations recorded by the source corpus.

Humans and animals are capable of learning a new behavior by observing others perform the skill just once. We consider the problem of allowing a robot to do the same -- learning from a raw video pixels of a human, even when there is substantial domain shift in the perspective, environment, and embodiment between the robot and the observed human. Prior approaches to this problem have hand-specified how human and robot actions correspond and often relied on explicit human pose detection systems. In this work, we present an approach for one-shot learning from a video of a human by using human and robot demonstration data from a variety of previous tasks to build up prior knowledge through meta-learning. Then, combining this prior knowledge and only a single video demonstration from a human, the robot can perform the task that the human demonstrated. We show experiments on both a PR2 arm and a Sawyer arm, demonstrating that after meta-learning, the robot can learn to place, push, and pick-and-place new objects using just one video of a human performing the manipulation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

In Robotics & Control

Citations in the corpus

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “Finn et al. (2017b) and Yu et al. (2018) both extend the Model-Agnostic Meta-Learning (MAML) method (Finn et al., 2017a), which we will brieﬂy review here before proceeding.”
CoMPS: Continual Meta Policy Search (2021, background): “…(Finn et al., 2017; Gupta et al., 2018b; Rothfuss et al., 2018; Clavera et al., 2018; Mendonca et al., 2019; Zintgraf et al., 2019; Mitchell et al., 2020), and training a critic to provide policy gradients (Sung et al., 2017; Houthooft et al., 2018; Yu et al., 2018; Chebotar et al., 2019).”
Sample Efficient Adaptive Text-to-Speech (2018, methodology): “This method has shown promise in robotics (Finn et al., 2017b; Yu et al., 2018).”
RLBench: The Robot Learning Benchmark & Learning Environment (2019, background): “2974707 there is now a large body of work looking at increasing the capabilities of robotic agents through the use of reinforcement learning [3], [4], meta-learning [5]–[7], multi-task learning [8],”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “Another powerful source of signal may come from imitation learning from demonstration, which several recent works have shown promise in using to produce real-time vision-based closed-loop policies [3], [9], [10], [13], [17], [18], [25], [26].”
Simitate: A Hybrid Imitation Learning Benchmark (2019, positive): “One shot imitation learning approaches [24], [25], [26] have recently gained popularity.”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, methodology): “Unlike prior work in few-shot imitation learning (Duan et al., 2017; Finn et al., 2017b; Yu et al., 2018; James et al., 2018; Paine et al., 2018), our method enables the agent to additionally improve upon trial-and-error experience.”
Modular meta-learning (2018, background): “Prominent examples of meta-learning in robotic domains are MAML [4] and follow-up work [5, 7].”
Playing hard exploration games by watching YouTube (2018, background): “Similarly, domain adaptive meta-learning [44] uses a shared policy network for addressing the domain gap for robotic tasks, though they require both ﬁrst person robot demonstrations and third person human demonstrations.”
Task-Embedded Control Networks for Few-Shot Imitation Learning (2018, background): “Parallel work has shown extensions to MIL that infer policies from human demonstration [32].”
Evolved Policy Gradients (2018, background): “(46) extended the model from (7) to incorporate a more elaborate learned loss function.”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “Many of these works adapt to the new scenario by condi59 tioning on a robot demonstration [1, 2], a video of a human [3, 4], a language instruction [23, 24], 60 or a goal image [21].”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, background): “Yu et al. (2018) and Goo & Niekum (2019) propose meta-learning-from-observation approaches that can learn from a single demonstration, but require training on a wide variety of similar tasks.”
Deep reinforcement learning: a survey (2020, methodology): “The improved methods of meta-RL (Finn et al., 2017b; Yu TH et al., 2018) enable the agents to learn rapidly from little data in new environments.”
R3M: A Universal Visual Representation for Robot Manipulation (2022, background): “Several prior works have explored using human video data in robot learning, for example to acquire goals [39, 40, 41], to learn visual dynamics models [42, 43, 44, 45], or to learn representations and rewards [19, 46, 47, 48, 49, 50].”
Federated Imitation Learning: A Privacy Considered Imitation Learning Framework for Cloud Robotic Systems with Heterogeneous Sensor Data (2019, background): “Some work has improved this issue, for example, oneshot imitation is achieved in [14].”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “…how robots can generalize to novel object instances [31, 34, 32, 35, 36], to tasks involving novel combinations of objects and skills [37, 38, 39, 40, 41], to new goals or language instructions [42, 43, 41, 44, 45, 46, 47], to tasks with novel semantic object categories [48, 49], and to…”
Learning Navigation Subroutines from Egocentric Videos (2019, neutral): “Works like [26, 27, 28] focus on imitation from visual data and learn a monolithic policy from expert data for the task at hand.”
Meta Learning via Learned Loss (2019, methodology): “In this context, we can divide related work on meta-learning into learning model parameters θ that can be easily adapted to new tasks [7, 18, 10, 35], learning optimizer policies h that transform parameters updates with respect to known loss or reward functions [16, 2, 14, 8, 17, 5], or learning los”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, background): “…Fakoor et al., 2020) and meta-imitation methods (Duan et al., 2017; Finn et al., 2017c; Huang et al., 2018; James et al., 2018; Paine et al., 2018; Yu et al., 2018; Huang et al., 2019; Zhou et al., 2020) also seek to speed up learning for new tasks by leveraging experience from previously seen…”