What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

By Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, Roberto Mart'in-Mart'in. Conference on Robot Learning, 2021

937 citations recorded by the source corpus.

Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch (offline) reinforcement learning, a lack of open-source human datasets and reproducible learning methods make assessing the state of the field difficult. In this paper, we conduct an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets of varying quality. Our study analyzes the most critical challenges when learning from offline human data for manipulation. Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation. We also highlight opportunities for learning from human datasets, such as the ability to learn proficient policies on challenging, multi-stage tasks beyond the scope of current reinforcement learning methods, and the ability to easily scale to natural, real-world manipulation scenarios where only raw sensory signals are available. We have open-sourced our datasets and all algorithm implementations to facilitate future research and fair comparisons in learning from human demonstration data. Codebase, datasets, trained models, and more available at https://arise-initiative.github.io/robomimic-web/

Subfields: Reinforcement Learning, Robotics & Control

View Paper: https://arxiv.org/abs/2108.03298

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models (2022, background): “However the performance of such methods is critically dependent on the quantity and breadth of instruction-labeled demonstration data that is available [32], and producing expert demonstrations of robot motion often requires expertise and time [33].”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, neutral): “Additional details about the policy training can be found in Appendix F.”
VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training (2022, background): “…as the data-driven paradigm of RL (Levine et al., 2020), is not necessarily data efﬁcient (Agarwal et al., 2021), requiring hundreds of thousands of samples even in low-dimensional simulated tasks, and requires a dense reward to operate most effectively (Mandlekar et al., 2021; Yu et al., 2022).”
Interactive Language: Talking to Robots in Real Time (2022, background): “Another challenge particular to large scale multi-operator collections is that typically not all data can be considered optimal [41], [42],”
Diffusion policy: Visuomotor policy learning via action diffusion (2023, positive): “In addition, we report the average of best-performing checkpoints for robomimic and Push-T tasks to be consistent with the evaluation methodology of their respective original papers (Mandlekar et al., 2021; Florence et al., 2021).”
BridgeData V2: A Dataset for Robot Learning at Scale (2023, background): “Assembling a large real-world dataset is time-consuming and expensive, so there has also been significant work on developing simulated environments and datasets for robotic manipulation [15, 40, 41, 42] and navigation [43, 44].”
One Step Diffusion via Shortcut Models (2024, neutral): “We compare to iterative denoising meth-ods IBC (Florence et al., 2022) and diffusion policy (Chi et al., 2023), as well as one-step methods LSTM-GMM (Mandlekar et al., 2021) and BET (Shafiullah et al., 2022).”
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (2023, methodology): “The RESNET-RNN [42] uses a ResNet as the visual backbone that encodes per-step visual observations and an LSTM as the temporal backbone to process a sequence of encoded visual information.”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, background): “They mainly focus on robotics [15, 44, 48, 40], some with a particular focus on human data [31, 32, 39].”
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (2024, neutral): “En-hancements to BC include incorporating history with various architectures [12, 47, 59, 77], new training objectives [10, 18, 35, 63, 104], regularization [71], motor primitives [7, 44, 55, 62, 64, 97], and data preprocessing [81].”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “We notice that such a data demand generally can be satisfied by both simulated robotics data (Mandlekar et al., 2021) and real robotics data (Dasari et al.”
Open-TeleVision: Teleoperation with Immersive Active Visual Feedback (2024, neutral): “If we distinguish the existing work with sources of demonstrations, we can classify them into learning from real-robot expert data [14, 10, 12, 59, 60, 61, 62, 50, 63, 1, 64, 51, 65], learning from play data [66, 67, 68], and learning from human demonstrations [67, 69, 70, 71, 72, 73, 74, 56].”