Habitat: A Platform for Embodied AI Research

By Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik and 2 more. 2019

2,003 citations recorded by the source corpus.

We present Habitat, a platform for research in embodied artificial intelligence (AI). Habitat enables training embodied agents (virtual robots) in highly efficient photorealistic 3D simulation. Specifically, Habitat consists of: (i) Habitat-Sim: a flexible, high-performance 3D simulator with configurable agents, sensors, and generic 3D dataset handling. Habitat-Sim is fast -- when rendering a scene from Matterport3D, it achieves several thousand frames per second (fps) running single-threaded, and can reach over 10,000 fps multi-process on a single GPU. (ii) Habitat-API: a modular high-level library for end-to-end development of embodied AI algorithms -- defining tasks (e.g., navigation, instruction following, question answering), configuring, training, and benchmarking embodied agents. These large-scale engineering contributions enable us to answer scientific questions requiring experiments that were till now impracticable or 'merely' impractical. Specifically, in the context of point-goal navigation: (1) we revisit the comparison between learning and SLAM approaches from two recent works and find evidence for the opposite conclusion -- that learning outperforms SLAM if scaled to an order of magnitude more experience than previous investigations, and (2) we conduct the first cross-dataset generalization experiments {train, test} x {Matterport3D, Gibson} for multiple sensors {blind, RGB, RGBD, D} and find that only agents with depth (D) sensors generalize across datasets. We hope that our open-source platform and these findings will advance research in embodied AI.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Embodied agents that use language are also being explored (Das et al., 2018; Savva et al., 2019; Szlam et al., 2019; Urbanek et al., 2019).”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “Many practitioners believe that multimodality holds the key to problems as varied as natural language understanding [5, 41], computer vision evaluation [23], and embodied AI [64].”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “Recently, many efforts have been put into creating more realistic environments with the goal to further advances in this area [38, 47, 48, 22, 44, 1].”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, neutral): “These benchmarks include language-driven navigation and embodied question answering, which have seen dramatic improvements in modeling thanks to environments like Matterport 3D [3, 11], AI2-THOR [25], and AI Habitat [44].”
Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, methodology): “0 is built using a tight integration of rendering from Habitat-Sim [3] and physics from the C++ Bullet library [4].”
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “…the color image. alistic synthetic datasets and interactive simulation environments that have spurred rapid progress towards the goal of holistic indoor scene understanding [6, 7, 9, 10, 14, 15, 19, 22, 24, 31, 33, 36, 37, 39, 43, 44, 45, 46, 49, 52, 61, 62, 63, 65, 70, 72, 75, 76, 79, 83, 84].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “Most of the larger-scale navigation-based simulators, such as AI2-THOR, iGibson and Habitat-Sim, tend to have navigation, atomic action and ROS [19, 34, 23] which enable them to provide better control and manipulation of objects in the environment while performing tasks such as Point Navigation or O”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “tions [36] and standardized object sets [37,38] that have played an important role in robotics research. Similarly, there exists a number of robotics simulation benchmarks including visual navigation [39,40,41,42,43], autonomous driving [44,45,46], grasping [47,48,49], single-task manipulation [50],”
AI2-THOR: An Interactive 3D Environment for Visual AI (2017, methodology): “Following AI2-THOR’s first release in 2017, a number of simulators have been developed, including iGibson 2.0 [18], Habitat 1.0 [30], Habitat 2.0 [32], ThreeDWorld [7], and SAPIEN [39].”
The Replica Dataset: A Digital Replica of Indoor Spaces (2019, methodology): “In addition, Replica is ‘Habitat-compatible’, i.e . can be natively used with AI Habitat [24] for training and testing embodied agents.”
SAPIEN: A SimulAted Part-Based Interactive ENvironment (2020, methodology): “Static environments, based on synthetic scenes [54] or real-world RGB-D scans [1] and reconstructions [41, 56, 42], are able to provide images that closely resembles reality, minimizing the domain gap in the visual aspect.”
Robust Navigation with Language Pretraining and Stochastic Sampling (2019, background): “The vision-and-language navigation (VLN) task, learning to navigate in visual environments based on natural language instructions, has attracted interest throughout the artiﬁcial intelligence research community (Hemachandra et al., 2015; Anderson et al., 2018; Chen et al., 2019; Savva et al., 2019).”
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects (2020, background): “Note that in contrast to [3,5], we increased the angles associated with turn-left and turn-right actions from 10◦ to 30◦ degrees due to the concerns described in [4].”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “Interaction in 3D environments Recent embodied AI work leverages simulated environments [45, 51, 57, 32, 22, 57] to build agents for interaction-heavy tasks like visual semantic planning [68], interactive question answering [25], instruction following [58], and object rearrangement [3, 30].”
iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, background): “The few simulation environments that include large scenes such as homes or ofﬁces either disable the possibility of interacting with the scene, focusing only on navigation (e.g. Habitat [18]), or use simpliﬁed modes of interaction (e.g. AI2Thor [19], VirtualHome [20]).”
The rise and potential of large language model based agents: a survey (2025, neutral): “Diﬀerent from the majority of RL algorithms [82,257–259], ﬁne-tuned by tasks in diverse forms and types, LLMs have showcased remarkable cross-task generalization capabilities [260,261].”
OpenRooms: An Open Framework for Photorealistic Indoor Scene Datasets (2020, background): “Several indoor virtual environments have also been proposed for robotics and embodied vision [54, 44, 55, 46, 38, 31].”
ReLMoGen: Integrating Motion Generation in Reinforcement Learning for Mobile Manipulation (2021, background): “Navigation-Only and Manipulation-Only Tasks: Point-Goal navigation [42, 43] and tabletop tasks [44] are mature robotic benchmarks.”
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, background): “, 2018) Low Ego Yes No No No AI Habitat (Savva et al., 2019) Low Ego Yes No No No VirtualHome-Social High/Low 3rd Person/Ego Yes Yes Yes Yes”