ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills

By Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Z. Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yuan Yao and 5 more. International Conference on Learning Representations, 2023

381 citations recorded by the source corpus.

Generalizable manipulation skills, which can be composed to tackle long-horizon and complex daily chores, are one of the cornerstones of Embodied AI. However, existing benchmarks, mostly composed of a suite of simulatable environments, are insufficient to push cutting-edge research works because they lack object-level topological and geometric variations, are not based on fully dynamic simulation, or are short of native support for multiple types of manipulation tasks. To this end, we present ManiSkill2, the next generation of the SAPIEN ManiSkill benchmark, to address critical pain points often encountered by researchers when using benchmarks for generalizable manipulation skills. ManiSkill2 includes 20 manipulation task families with 2000+ object models and 4M+ demonstration frames, which cover stationary/mobile-base, single/dual-arm, and rigid/soft-body manipulation tasks with 2D/3D-input data simulated by fully dynamic engines. It defines a unified interface and evaluation protocol to support a wide range of algorithms (e.g., classic sense-plan-act, RL, IL), visual observations (point cloud, RGBD), and controllers (e.g., action type and parameterization). Moreover, it empowers fast visual input learning algorithms so that a CNN-based policy can collect samples at about 2000 FPS with 1 GPU and 16 processes on a regular workstation. It implements a render server infrastructure to allow sharing rendering resources across all environments, thereby significantly reducing memory usage. We open-source all codes of our benchmark (simulator, environments, and baselines) and host an online challenge open to interdisciplinary researchers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/634127f09be5119f4d29a175866f11b14b3821e0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (2024, background): “RoboCasa is one of the few frameworks to feature photorealistic rendering [21, 40, 27, 10, 6] and multiple robot embodiments [21, 26, 27, 6, 34].”
Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments (2023, background): “Frameworks that rely on CPU-vectorization ([2], [4], [10]) show an increase in throughput with the number of environments.”
RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation (2025, positive): “…articulated objects from PartNet-Mobility [40, 28], ManiSkill2 supplies more than four million demonstration frames across twenty task families [14], Meta-World [42] offers fifty standardized manipulation tasks for multi-task and meta-reinforcement learning, CALVIN pairs long-horizon,…”
TD-MPC2: Scalable, Robust World Models for Continuous Control (2023, methodology): “We evaluate TD-MPC 2 across a total of 104 diverse continuous control tasks spanning 4 task domains: DMControl (Tassa et al., 2018), Meta-World (Yu et al., 2019), ManiSkill2 (Gu et al., 2023), and MyoSuite (Caggiano et al., 2022).”
Evaluating Real-World Robot Manipulation Policies in Simulation (2024, positive): “That is, if one policy performs better than another in real-world 1] , higher is better) for assessing the correlation between policy performances in real-world and simulation, as well as the overall quality of simulated evaluation pipelines.”
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (2023, background): “Robot Learning Benchmarks A variety of robot learning benchmarks have been proposed to address challenges in meta learning (MetaWorld [73]), causality learning (CausalWorld [1]), multi-task learning [27, 35], policy generalization to unseen objects [47, 24], and compositional learning [44].”