RoboTHOR: An Open Simulation-to-Real Embodied AI Platform

By Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford and 3 more. 2020

319 citations recorded by the source corpus.

Visual recognition ecosystems (e.g. ImageNet, Pascal, COCO) have undeniably played a prevailing role in the evolution of modern computer vision. We argue that interactive and embodied visual AI has reached a stage of development similar to visual recognition prior to the advent of these ecosystems. Recently, various synthetic environments have been introduced to facilitate research in embodied AI. Notwithstanding this progress, the crucial question of how well models trained in simulation generalize to reality has remained largely unanswered. The creation of a comparable ecosystem for simulation-to-real embodied AI presents many challenges: (1) the inherently interactive nature of the problem, (2) the need for tight alignments between real and simulated worlds, (3) the difficulty of replicating physical conditions for repeatable experiments, (4) and the associated cost. In this paper, we introduce RoboTHOR to democratize research in interactive and embodied visual AI. RoboTHOR offers a framework of simulated environments paired with physical counterparts to systematically explore and overcome the challenges of simulation-to-real transfer, and a platform where researchers across the globe can remotely test their embodied models in the physical world. As a first benchmark, our experiments show there exists a significant gap between the performance of models trained in simulation when they are tested in both simulations and their carefully constructed physical analogs. We hope that RoboTHOR will spur the next stage of evolution in embodied computer vision.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “AI2-THOR, which are primarily designed for visual navigation, are also equipped with robotic embodiment for ease of deployment in their real-world counterparts such as iGibson’s Castro [37] and RoboTHOR [31] respectively.”
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects (2020, methodology): “RoboTHOR [8] is a platform to develop and test embod-ied AI agents with corresponding environments in simulation and the physical world (Fig.”
AllenAct: A Framework for Embodied AI Research (2020, methodology): “…focused on manipulation (such as RL-Bench [5], Sapien [6], and Meta-world [7]), using advanced physics simulations (such as MuJoCo [8] and ThreeDWorld [9]), and also physical counterparts to simulation environments (RoboTHOR [10] and iGibson [11]) to enable research in simulation-to-real transfer.”
Domain Generalization: A Survey (2021, background): “ment often suffer from overfitting and hence generalize poorly to unseen environments [92], [122], [123], [124].”
Visual Room Rearrangement (2021, methodology): “4, follows the same basic structure as is commonly employed within Embod-ied AI tasks [38, 47, 14, 45, 23]: a combination of a convolutional neural network to process input egocentric images, a collection of embedding layers to encode discrete inputs, and an RNN to enable the agent to reason…”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, methodology): “RoboTHOR [13] houses use wall panels and floors with very specific textures.”
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI (2021, methodology): “We also compute the metrics for the RoboTHOR [14] dataset which is synthetic but based on real-world layouts.”
Online Continual Learning For Interactive Instruction Following Agents (2024, background): “…vision, natural language processing, and embodied AI have led to various benchmarks for robotic agents, encompassing navigation (Savva et al., 2019; Deitke et al., 2020; Anderson et al., 2018; Krantz et al., 2020), object interaction (Zhu et al., 2017; Misra et al., 2017; Weihs et al., 2021;…”
Evaluating Real-World Robot Manipulation Policies in Simulation (2024, unknown): “Furthermore, building full-fidelity digital twins demands extensive time and resources, typically requiring digital artists to manually craft object geometries and materials [14, 59].”
How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget (2020, background): “agent variants for a fixed number of samples – 75 million steps, a high but feasible number of samples [5, 10, 21, 22, 31].”
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark Towards Physically Realistic Embodied AI (2021, background): “Recently, the development of 3D interactive environments, including AI2-THOR[25], HoME [44], VirtualHome [27], Habitat [31], Gibson [47], Deepmind Lab [4], RoboTHOR [14], SAPIEN [48], and TDW [15] have facilitated novel algorithms in visual navigation [52, 31, 28], visual-language navigation [1, 41,”
Simple but Effective: CLIP Embeddings for Embodied AI (2021, methodology): “Simulators are photo-realistic, simulate physics, and enable interaction between agents and their environments [7, 8, 15, 16, 25, 26, 30, 36].”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, methodology): “In different navigation benchmarks (e.g., Habitat [59] and RoboTHOR [17]), an agent could experience drastically different object and layout distributions, visual appearance and style, as well as different noise patterns from sensors and actuators.”