The MAGICAL Benchmark for Robust Imitation

By Sam Toyer, Rohin Shah, Andrew Critch, Stuart J. Russell. 2020

41 citations recorded by the source corpus.

Imitation Learning (IL) algorithms are typically evaluated in the same environment that was used to create demonstrations. This rewards precise reproduction of demonstrations in one particular environment, but provides little information about how robustly an algorithm can generalise the demonstrator's intent to substantially different deployment settings. This paper presents the MAGICAL benchmark suite, which permits systematic evaluation of generalisation by quantifying robustness to different kinds of distribution shift that an IL algorithm is likely to encounter in practice. Using the MAGICAL suite, we confirm that existing IL algorithms overfit significantly to the context in which demonstrations are provided. We also show that standard methods for reducing overfitting are effective at creating narrow perceptual invariances, but are not sufficient to enable transfer to contexts that require substantially different behaviour, which suggests that new approaches will be needed in order to robustly generalise demonstrator intent. Code and data for the MAGICAL suite is available at https://github.com/qxcv/magical/.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Robotics & Control

Citations in the corpus

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, background): “Prior works are either limited to studying simple 2D environments [23] or using data generated from hard-coded policies [24, 25].”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “…Dasari & Gupta, 2020; Aceituno et al., 2021; Zhao et al., 2022), rearrangement (Weihs et al., 2021; Szot et al., 2021; Liu et al., 2021; Ehsani et al., 2021; Gan et al., 2021; Stengel-Eskin et al., 2022), and reasoning (Gupta et al., 2019; Ahmed et al., 2021; Toyer et al., 2020; Lim et al., 2021).”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, background): “They mainly focus on robotics [15, 44, 48, 40], some with a particular focus on human data [31, 32, 39].”
XIRL: Cross-embodiment Inverse Reinforcement Learning (2021, methodology): “1 X-MAGICAL Benchmark We introduce a cross-embodiment imitation learning benchmark, X-MAGICAL, which is based on the imitation learning benchmark MAGICAL [33], implemented on top of the physics engine PyMunk [34].”
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, unknown): “Our main experiments (Tables II and III) demonstrate that our method adheres to the scaling law: as model size increases, the average success rate across tasks improves accordingly.”
Diffusion policy: Visuomotor policy learning via action diffusion (2023, unknown): “The simplest form of explicit policies maps from world state or observation directly to action (Pomerleau, 1988; Zhang et al., 2018; Florence et al., 2020; Ross et al., 2011; Toyer et al., 2020; Rahmatizadeh et al., 2018); Bojarski et al., 2016).”