ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

By Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, J. Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano and 13 more. 2020

385 citations recorded by the source corpus.

We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. With TDW, users can simulate high-fidelity sensory data and physical interactions between mobile agents and objects in a wide variety of rich 3D environments. TDW has several unique properties: 1) realtime near photo-realistic image rendering quality; 2) a library of objects and environments with materials for high-quality rendering, and routines enabling user customization of the asset library; 3) generative procedures for efficiently building classes of new environments 4) high-fidelity audio rendering; 5) believable and realistic physical interactions for a wide variety of material types, including cloths, liquid, and deformable objects; 6) a range of "avatar" types that serve as embodiments of AI agents, with the option for user avatar customization; and 7) support for human interactions with VR devices. TDW also provides a rich API enabling multiple agents to interact within a simulation and return a range of sensor and physics data representing the state of the world. We present initial experiments enabled by the platform around emerging research directions in computer vision, machine learning, and cognitive science, including multi-modal physical scene understanding, multi-agent interactions, models that "learn like a child", and attention studies in humans and neural networks. The simulation platform will be made publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, 3D Vision & Pose, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

In Robotics & Control

Citations in the corpus

Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “How are interactive simulators built today? Either by relying on game engines [7, 51, 52] or via a ‘homebrew’ integration of existing rendering and physics libraries [8, 35, 37, 53].”
Continuous Scene Representations for Embodied AI (2022, background): “Recently, various embodied tasks such as navigation [5,8,37,54,56,63], instruction following [1,47], embodied question answering [15,23], object manipulation [18,21,22,46], and room rearrangement [4,52] have been of interest.”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “This paper will focus on the following nine popular embodied AI simulators that were developed over the past four years: DeepMind Lab [18], AI2-THOR [19], CHALET [20], VirtualHome [21], VRKitchen [22], Habitat-Sim [23], iGibson [24], SAPIEN [25], and ThreeDWorld [26].”
AllenAct: A Framework for Embodied AI Research (2020, methodology): “…focused on manipulation (such as RL-Bench [5], Sapien [6], and Meta-world [7]), using advanced physics simulations (such as MuJoCo [8] and ThreeDWorld [9]), and also physical counterparts to simulation environments (RoboTHOR [10] and iGibson [11]) to enable research in simulation-to-real transfer.”
Objaverse: A Universe of Annotated 3D Objects (2022, background): “To put the scale of OBJAVERSE-Interiors in perspective, the number of scenes in OBJAVERSE-Interiors is significantly larger than the 400 or so existing hand-built interactive embodied AI scenes [23, 39, 42, 72].”
Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments (2023, background): “c) Scene designing and asset handling: Frameworks support scene creation procedurally [1], [2], [11], via mesh scans [6], [7] or through game-engine style interfaces [9], [26].”
PIP: Physical Interaction Prediction via Mental Imagery with Span Selection (2021, background): “Physion [40] presents a dataset which comprises eight physical interaction scenarios in the ThreeDWorld simulator [8].”
ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills (2023, methodology): “(7)For TDW (Gan et al., 2020), we refer to its Transport Challenge (Gan et al.”
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning (2021, background): “Recently, various benchmarks (Riochet et al., 2018; Bakhtin et al., 2019; Girdhar & Ramanan, 2020; Baradel et al., 2020; Gan et al., 2020) have been proposed to study dynamics and reasoning in physical scenes.”
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark Towards Physically Realistic Embodied AI (2021, positive): “We build our challenge using ThreeDWorld (TDW) [15], an interactive physical simulation platform.”
Curious Representation Learning for Embodied Intelligence (2021, result): “In contrast to prior work, which focuses on synthetic game environments, we study representation learning in photorealistic 3D environments [16, 33, 48].”
Physion: Evaluating Physical Prediction from Vision in Humans and Machines (2021, methodology): “We used the ThreeDWorld simulator (TDW), a Unity3D-based environment [24], to create eight physical scenarios out of simple objects that incorporate diverse physical phenomena (Fig.”
Building Cooperative Embodied Agents Modularly with Large Language Models (2023, background): “In Published as a conference paper at ICLR 2024”
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, background): “, 2016) High/Low 3rd Person/Ego No No No Yes ThreeDWorld (Gan et al., 2020) High/Low 3rd Person/Ego Yes No No Yes VRKitchen (Gao et al.”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, background): “Previous work has explored 3D-rendered datasets [18, 73] and simulation environments with physically realistic engines [11, 15, 16].”
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks (2021, neutral): “Simulation environments with (mostly) kinodynamic simulation: In the last years, the robotics and AI communities have presented several impressive simulation environments and benchmarks: iGibson [3, 1], Habitat AI [5], AI2Thor (and variants) [15, 6], ThreeDWorld [7], Sapien [4], Robo-suite [9], Virt”
iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, methodology): “Gibson [25] Habitat [18] Sapien [15] AI2Thor [19] VirtualH [20] TDW [26] Provided Large Scenes (RS,AS)”