MimicPlay: Long-Horizon Imitation Learning by Watching Human Play

By Linxi (Jim) Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, Anima Anandkumar. Conference on Robot Learning, 2023

343 citations recorded by the source corpus.

Imitation learning from human demonstrations is a promising paradigm for teaching robots manipulation skills in the real world. However, learning complex long-horizon tasks often requires an unattainable amount of demonstrations. To reduce the high data requirement, we resort to human play data - video sequences of people freely interacting with the environment using their hands. Even with different morphologies, we hypothesize that human play data contain rich and salient information about physical interactions that can readily facilitate robot policy learning. Motivated by this, we introduce a hierarchical learning framework named MimicPlay that learns latent plans from human play data to guide low-level visuomotor control trained on a small number of teleoperated demonstrations. With systematic evaluations of 14 long-horizon manipulation tasks in the real world, we show that MimicPlay outperforms state-of-the-art imitation learning methods in task success rate, generalization ability, and robustness to disturbances. Code and videos are available at https://mimic-play.github.io

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/a68db57f08f6d72c0d3b22d451d2606dca880f94

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (2024, background): “…also focus on multi-task or few-shot imitation learning, [25, 27, 30, 34, 46, 50, 88, 102], language-conditioned imitation learning [12, 47, 82, 83], imitation from play data [21, 57, 74, 89], using human videos [16, 24, 29, 60, 69, 80, 84, 96], and using task-specific structures [49, 83, 103].”
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (2023, unknown): “We compute the multi-modal distribution over manipulation actions using a Gaussian-Mixture-Model (GMM) based output head [8, 42, 68].”
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators (2023, unknown): “Finally, a promising direction is learning directly from human videos [52], [53], [54], [55], [56], [57].”
Open-TeleVision: Teleoperation with Immersive Active Visual Feedback (2024, unknown): “If we distinguish the existing work with sources of demonstrations, we can classify them into learning from real-robot expert data [14, 10, 12, 59, 60, 61, 62, 50, 63, 1, 64, 51, 65], learning from play data [66, 67, 68], and learning from human demonstrations [67, 69, 70, 71, 72, 73, 74, 56].”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “Notably, MimicPlay [118] presents a hierarchical imitation learning algorithm that learns high-level plans in latent spaces from human play data and low-level motor commands from a small number of teleoperated demonstrations.”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “MIMICPLAY [125] addresses domain discrepancies by integrating human play data and robot teleoperation data.”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “The learned cost functions are then used by reinforcement learning [102, 99, 107], imitation learning [97, 96], or trajectory optimization [95, 35] to generate robot actions.”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “Alternatively, some works focus on sub-aspects of the problem such as learning transferable reward functions [17, 44–48], goals [49, 50], dynamics models [51], or visual representations [52–59] from human video data.”
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots (2024, methodology): “To infer action data from passive human video, previous works resort to hand pose detectors [44, 1, 38, 28], or combining human videos with in-domain teleoperated robot data to predict actions [33, 20, 34, 28].”