Simitate: A Hybrid Imitation Learning Benchmark

By Raphael Memmesheimer, Ivanna Mykhalchyshyna, Viktor Seib, Dietrich Paulus. 2019

20 citations recorded by the source corpus.

We present Simitate — a hybrid benchmarking suite targeting the evaluation of approaches for imitation learning. A dataset containing 1938 sequences where humans perform daily activities in a realistic environment is presented. The dataset is strongly coupled with an integration into a simulator. RGB and depth streams with a resolution of $960 \times 540$ at 30Hz and accurate ground truth poses for the demonstrator’s hand, as well as the object in 6 DOF at 120Hz are provided. Along with our dataset we provide the 3D model of the used environment, labeled object images and pre-trained models. A benchmarking suite that aims at fostering comparability and reproducibility supports the development of imitation learning approaches. Further, we propose and integrate evaluation metrics on assessing the quality of effect and trajectory of the imitation performed in simulation. Simitate is available on our project website: https://agas.uni-koblenz.de/simitate/.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, background): “Simitate [69] is an imitation learning benchmark suite consisting of real-world motion trajectories collected by humans, and carefully translated into simulation using extensive sensor instrumentation in the real world.”
The MAGICAL Benchmark for Robust Imitation (2020, background): “We note that there also exist dedicated IL benchmarks [30, 26], but they are aimed at solving challenging robotics tasks rather than evaluating generalisation directly.”
RLBench: The Robot Learning Benchmark & Learning Environment (2019, methodology): “Although Simitate offers the beneﬁt of being partially a real-world dataset, the addition of new tasks requires time-consuming calibration and motion capturing; our system on the other hand sacriﬁces the real-world aspect, but in exchange we receive the ability generate a diverse range of tasks in a”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, background): “They mainly focus on robotics [15, 44, 48, 40], some with a particular focus on human data [31, 32, 39].”