SAPIEN: A SimulAted Part-Based Interactive ENvironment

By Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Bernie Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang and 4 more. 2020

843 citations recorded by the source corpus.

Building home assistant robots has long been a goal for vision and robotics researchers. To achieve this task, a simulated environment with physically realistic simulation, sufficient articulated objects, and transferability to the real robot is indispensable. Existing environments achieve these requirements for robotics simulation with different levels of simplification and focus. We take one step further in constructing an environment that supports household tasks for training robot learning algorithm. Our work, SAPIEN, is a realistic and physics-rich simulated environment that hosts a large-scale set of articulated objects. SAPIEN enables various robotic vision and interaction tasks that require detailed part-level understanding.We evaluate state-of-the-art vision algorithms for part detection and motion attribute recognition as well as demonstrate robotic interaction tasks using heuristic approaches and reinforcement learning algorithms. We hope that SAPIEN will open research directions yet to be explored, including learning cognition through interaction, part motion discovery, and construction of robotics-ready simulated game environment.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Experience Grounds Language (2020, background): “The last few years have seen massive advances in both high ﬁdelity simulators for robotics (Todorov et al., 2012; Coumans and Bai, 2016–2019; NVIDIA, 2019; Xiang et al., 2020) and the cost and availability of commodity hardware (Fitzgerald, 2013; Campeau-Lecours et al., 2019; Murali et al., 2019).”
Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “How are interactive simulators built today? Either by relying on game engines [7, 51, 52] or via a ‘homebrew’ integration of existing rendering and physics libraries [8, 35, 37, 53].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “Most game-based scene simulators [13], [16], [19], [20] provides both fine-grained object manipulation capabilities and symbolic interaction capabilities (e.”
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, background): “Since most existing articulated object datasets contain neither twohands nor human bodies [41, 43, 76], and having a human in the scene is a realistic setting, we believe that ARCTIC brings additional challenges of heavy occlusion and dynamic manipulation to the depth community.”
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation (2020, methodology): “These environments include 66 AI2-THOR[21], HoME[40], VirtualHome[31], Habitat[33], Gibson[42], iGibson [41], Sapien [43] 67 PyBullet [9], MuJuCo [36], and Deepmind Lab [6].”
Visual Room Rearrangement (2021, background): “There is a general trend towards interactive tasks [50, 41, 49].”
Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery (2021, methodology): “We evaluate our approach on eight object categories from the PartNet-Mobility dataset [9, 29, 49] and a ninth multilink category, which we conﬁgure with three links.”
Evaluating Real-World Robot Manipulation Policies in Simulation (2024, neutral): “5) Are our results applicable for a different physics simulator ?”
Instruction-driven history-aware policies for robotic manipulations (2022, background): “The emergence of robotic simulators, such as Gym [52], manipulaTHOR [53], dm control [54], Sapien [55], CausalWorld [56], and RL-Bench [19], also greatly accelerated the development of manipulation methods.”
BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments (2021, positive): “We either directly acquire them from the PartNet-Mobility Dataset [77] or acquire non-articulated models from TurboSquid, manually segment the models into parts, and then create the articulation in the URDF ﬁles.”
Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments (2023, methodology): “While recent frameworks [7], [11], [12] include physically-based renderers, they mainly support camera-based sensors (RGB, depth).”
iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, methodology): “The object models are extended from open-source datasets [21, 22, 15] enriched with annotations of material and dynamic properties.”
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark Towards Physically Realistic Embodied AI (2021, background): “Recently, the development of 3D interactive environments, including AI2-THOR[25], HoME [44], VirtualHome [27], Habitat [31], Gibson [47], Deepmind Lab [4], RoboTHOR [14], SAPIEN [48], and TDW [15] have facilitated novel algorithms in visual navigation [52, 31, 28],”
AllenAct: A Framework for Embodied AI Research (2020, methodology): “Platforms such as RLBench [5], Sapien [6], and Meta-World [7] focus on manipulation tasks, while [46, 47] enable studying the task of grasping.”
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos (2021, positive): “Our annotations are based on 3D mesh models from Xiang et al. [33] which we use to represent objects with moving parts that articulate along a speciﬁc motion axis and origin.”
BEHAVIOR-1K: A Benchmark for Embodied AI with 1, 000 Everyday Activities and Realistic Simulation (2022, background): “On the other hand, household rearrangement benchmarks such as Habitat 2.0 HAB [26], TDW Transport [19], and SAPIEN ManiSkill [54, 55] support realistic action execution and accurate physics simulation for rigid bodies, but only include a handful of tasks.”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, background): “0 [67], iGibson [64], Sapien [73], and ThreeDWorld [14], which feature a wide range of household objects and scene-level interaction tasks.”
Objaverse: A Universe of Annotated 3D Objects (2022, background): “Decomposing 3D objects into parts has led to a flurry of research in the past few years, including work in learning robotic grasping policies [84, 86], 3D semantic segmentation [50], and shape generation [49].”