One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL

By T. Paine, Sergio Gomez Colmenarejo, Ziyun Wang, Scott E. Reed, Yusuf Aytar, Tobias Pfaff, Matthew W. Hoffman, Gabriel Barth-Maron, Serkan Cabi, David M. Budden and 1 more. 2018

26 citations recorded by the source corpus.

Humans are experts at high-fidelity imitation -- closely mimicking a demonstration, often in one attempt. Humans use this ability to quickly solve a task instance, and to bootstrap learning of new tasks. Achieving these abilities in autonomous agents is an open problem. In this paper, we introduce an off-policy RL algorithm (MetaMimic) to narrow this gap. MetaMimic can learn both (i) policies for high-fidelity one-shot imitation of diverse novel skills, and (ii) policies that enable the agent to solve tasks more efficiently than the demonstrators. MetaMimic relies on the principle of storing all experiences in a memory and replaying these to learn massive deep neural network policies by off-policy RL. This paper introduces, to the best of our knowledge, the largest existing neural networks for deep RL and shows that larger networks with normalization are needed to achieve one-shot high-fidelity imitation on a challenging manipulation task. The results also show that both types of policy can be learned from vision, in spite of the task rewards being sparse, and without access to demonstrator actions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Simitate: A Hybrid Imitation Learning Benchmark (2019, positive): “One shot imitation learning approaches [24], [25], [26] have recently gained popularity.”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, methodology): “Unlike prior work in few-shot imitation learning (Duan et al., 2017; Finn et al., 2017b; Yu et al., 2018; James et al., 2018; Paine et al., 2018), our method enables the agent to additionally improve upon trial-and-error experience.”
Learning Latent Plans from Play (2019, neutral): “In contrast to prior work that uses reinforcement learning (Paine et al. [31]), it does not require any reward function or costly RL phase.”
Playing hard exploration games by watching YouTube (2018, background): “Other approaches include interacting with the environment before introducing the expert demonstrations [38, 31] and goal conditioned policies for high ﬁdelity imitation [29], although these papers typically do not assume domain gap or operate in sparse reward settings.”
Making Efficient Use of Demonstrations to Solve Hard Exploration Problems (2019, neutral): “Veˇcerík et al. (2017), Merel et al. (2017) and Paine et al. (2018) have demonstrated similar results in robotics.”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “Multiple prior imitation learning works have achieved different forms of generalization, includ56 ing one-shot generalization to novel objects [1, 2, 3, 4, 18], to novel object configurations [19], 57 and to novel goal configurations [6, 7, 20], as well as zero-shot generalization to new objects [5]”
An Introduction to Deep Reinforcement Learning (2018, unknown): “The agent can then act based on new unseen trajectories of the teacher, with the objective that is can generalize sufficiently well to perform new tasks (Paine et al., 2018).”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, background): “…et al., 2020; Fakoor et al., 2020) and meta-imitation methods (Duan et al., 2017; Finn et al., 2017c; Huang et al., 2018; James et al., 2018; Paine et al., 2018; Yu et al., 2018; Huang et al., 2019; Zhou et al., 2020) also seek to speed up learning for new tasks by leveraging experience…”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, unknown): “, 2021), one-shot imitation (Paine et al., 2018; Huang et al., 2019; Dasari & Gupta, 2020; Aceituno et al., 2021; Zhao et al., 2022), rearrangement (Weihs et al.”