Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation

By Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, Abhinav Kumar Gupta. 2018

133 citations recorded by the source corpus.

In recent years, we have seen an emergence of data-driven approaches in robotics. However, most existing efforts and datasets are either in simulation or focus on a single task in isolation such as grasping, pushing or poking. In order to make progress and capture the space of manipulation, we would need to collect a large-scale dataset of diverse tasks such as pouring, opening bottles, stacking objects etc. But how does one collect such a dataset? In this paper, we present the largest available robotic-demonstration dataset (MIME) that contains 8260 human-robot demonstrations over 20 different robotic tasks (this https URL). These tasks range from the simple task of pushing objects to the difficult task of stacking household objects. Our dataset consists of videos of human demonstrations and kinesthetic trajectories of robot demonstrations. We also propose to use this dataset for the task of mapping 3rd person video features to robot trajectories. Furthermore, we present two different approaches using this dataset and evaluate the predicted robot trajectories against ground-truth trajectories. We hope our dataset inspires research in multiple areas including visual imitation, trajectory prediction, and multi-task robotic learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

RoboNet: Large-Scale Multi-Robot Learning (2019, background): “Our work is inspired by this idea: while prior robotic manipulation methods and datasets [5, 46, 47, 26, 48, 7, 49] generally consider a single robot at a time, our dataset includes 7 different robots and data from 4 different institutions, with dozens of backgrounds and hundreds of viewpoints.”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “There is a long history of robotics benchmarks [28], datasets [29, 30, 31, 32, 33, 34, 35], competitions [36] and standardized object sets [37, 38] that have played an important role in robotics research.”
BridgeData V2: A Dataset for Robot Learning at Scale (2023, background): “Many existing robot datasets contain only one or a few environments and tasks [3, 4, 5], meaning a researcher would need to exactly replicate a scene from the data to use them for robot learning.”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, background): “A number of prior works in robotics have also focused on collecting datasets containing demonstrations or trials that illustrate a variety of different tasks [55, 6, 70, 59, 22].”
R3M: A Universal Visual Representation for Robot Manipulation (2022, background): “Many of these works focus on collecting and learning from robot data itself [55, 56, 7, 8, 9, 10, 57].”
Offline Reinforcement Learning from Images with Latent Space Models (2020, background): “, 2020), recommendation systems (Harper and Konstan, 2015), and robotic manipulation (Finn and Levine, 2017; Sharma et al., 2018; Dasari et al., 2019; Mandlekar et al., 2019), typically in image (or video) format.”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “Prior work learns dynamics models for behavior cloning [56, 66, 43, 53, 52] or crafts reward functions that encourage visiting expert states [1, 16, 54, 35].”
RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (2024, unknown): “…the action space, which exacerbates the data scarcity (Zoll-ner et al., 2004; Smith et al., 2012; Lioutikov et al., 2016; Stepputtis et al., 2022) and the multi-modal behavior (Colom ´ e & Torras, 2018; 2020; Figueroa & Billard, 2017; Sharma et al., 2018; Xie et al., 2020; Franzese et al., 2023).”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, background): “They mainly focus on robotics [15, 44, 48, 40], some with a particular focus on human data [31, 32, 39].”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “Even the largest data collection efforts still end up with datasets that are a fraction of the size and diversity of benchmark datasets in vision (5-18M) [4, 5] and NLP (1.5B-4.”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “Another line of work in imitation learning is to train policies to follow the expert demonstrations [43, 52, 49, 71, 68, 72, 74, 21, 51, 73].”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, unknown): “Broadly, these can be categorized into datasets collected autonomously via scripted and semi-random behaviors or learned agents [3, 8, 20, 26, 27, 32, 40], and datasets collected via human teleoperation [1, 2, 13, 14, 24, 36, 50, 59].”
Diffusion policy: Visuomotor policy learning via action diffusion (2023, unknown): “Another approach is to combine Categorical and Gaussian distributions to represent continuous multimodal distributions via the use of MDNs (Bishop, 1994; Mandlekar et al., 2021) or clustering with offset prediction (Sha ﬁ ullah et al., 2022; Sharma et al., 2018).”