AI2-THOR: An Interactive 3D Environment for Visual AI

By Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu and 3 more. 2017

1,486 citations recorded by the source corpus.

We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at this http URL AI2-THOR consists of near photo-realistic 3D indoor scenes, where AI agents can navigate in the scenes and interact with objects to perform tasks. AI2-THOR enables research in many different domains including but not limited to deep reinforcement learning, imitation learning, learning by interaction, planning, visual question answering, unsupervised representation learning, object detection and segmentation, and learning models of cognition. The goal of AI2-THOR is to facilitate building visually intelligent models and push the research forward in this domain.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “Recently, many efforts have been put into creating more realistic environments with the goal to further advances in this area [38, 47, 48, 22, 44, 1].”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, methodology): “, 2020), set in the THOR simulator (Kolve et al., 2017), is a benchmark for learning to complete embodied household tasks using natural language instructions and egocentric visual observations.”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “These benchmarks include language-driven navigation and embodied question answering, which have seen dramatic improvements in modeling thanks to environments like Matterport 3D [3, 11], AI2-THOR [25], and AI Habitat [44].”
Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “How are interactive simulators built today? Either by relying on game engines [7, 51, 52] or via a ‘homebrew’ integration of existing rendering and physics libraries [8, 35, 37, 53].”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, background): “Learning to navigate in a photorealistic home environment based on natural language instructions has attracted increasing research interest [23, 14, 7, 3, 6], as it provides insight into core scientific questions about multimodal representations.”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, methodology): “ on embodied question answering [16,18]. Navigation based simulators Our simulator is related to existing 3D RL environments based on game engines, such as ViZDoom [27], DeepMind Lab [7] and AI2-THOR [30], as well as a number of newer environments developed concurrently including HoME [10], House3D ”
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “…[9, 10, 79] S+I Matterport3D [7, 15, 61, 62, 79] S+I ScanNet [19] S+I Gibson [62, 79] Replica [62, 72] S+I Synthetic (artist-created) AI2-THOR [39] S+I ARAP [14] I D+R SceneNet-RGBD [52] S+I PBRS [70, 83] S+I CGIntrinsics [45, 70] D InteriorNet [43] S+I D Jiang et al. [36] S+I RobotriX [24]…”
Embodied Question Answering (2017, background): “[32] studied the task of EmbodiedQA in the AI2-THOR environment [33].”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “The training of their model requires the features output from the pretrained ResNet - 50 (He et al., 2016), and is performed in an indoor simulator (Kolve et al., 2017) where each new room is regarded as a new scene for which several scene-speciﬁc layers are added as another output head of the…”
Continuous Scene Representations for Embodied AI (2022, methodology): “To train the CSR encoder f CSR , which embeds nodes and edges, we generate a dataset using AI2-THOR [30].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “Most of the larger-scale navigation-based simulators, such as AI2-THOR, iGibson and Habitat-Sim, tend to have navigation, atomic action and ROS [19, 34, 23] which enable them to provide better control and manipulation of objects in the environment while performing tasks such as Point Navigation or O”
The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation (2018, methodology): “SceneNet RGB-D [9], [10] and AI2-THOR [1], [11] are two recently proposed synthetic 3D scenes datasets.”
robosuite: A Modular Simulation Framework and Benchmark for Robot Learning (2020, background): “In recent years, advances in physics-based simulations and graphics have led to a series of simulated platforms and toolkits [1, 14, 8, 2, 16] that have accelerated scientific progress on robotics and embodied AI.”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “Similarly, there exists a number of robotics simulation benchmarks including visual navigation [39, 40, 41, 42, 43], autonomous driving [44, 45, 46], grasping [47, 48, 49], single-task manipulation [50], among others.”
IQA: Visual Question Answering in Interactive Environments (2017, methodology): “Towards this end, we present the Interactive Question Answering Dataset (IQUAD V1) built upon AI2-THOR [35], a photo-realistic customizable simulation environment for indoor scenes integrated with the Unity [1] physics engine.”
Habitat: A Platform for Embodied AI Research (2019, background): “The availability of large-scale 3D scene datasets [5, 24, 8] and community interest in active vision tasks led to a recent surge of work that resulted in the development of a variety of simulation platforms for indoor environments [17, 7, 13, 22, 26, 3, 27, 28, 21].”
Learning to Speak and Act in a Fantasy Text Adventure Game (2019, background): “A number of visual, rather than text, platforms have been proposed such as House3D (Wu et al., 2018b), HoME (Brodeur et al., 2017), MINOS (Savva et al., 2017), Matterport3D (Chang et al., 2017) and AI2-THOR (Kolve et al., 2017), and the Minecraft MALMO project (Johnson et al., 2016), but they typica”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “In the past two years, a plethora of tasks and evaluation protocols [32, 22, 34, 45, 2] have been proposed as summarized in [1].”
BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning (2018, background): “Currently, most general-purpose simulation frameworks do not feature language, such as PycoLab (DeepMind, 2017), MazeBase (Sukhbaatar et al., 2015), Gazebo (Koenig and Howard, 2004), Viz-Doom (Kempka et al., 2016), DM-30 (Espeholt et al., 2018), and AI2-Thor (Kolve et al., 2017).”
Vision-and-Dialog Navigation (2019, background): “, [34]) or using virtual reality technology may facilitate easier transfer to physical robot platforms.”