Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI

By Santhosh K. Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X. Chang and 3 more. 2021

764 citations recorded by the source corpus.

We present the Habitat-Matterport 3D (HM3D) dataset. HM3D is a large-scale dataset of 1,000 building-scale 3D reconstructions from a diverse set of real-world locations. Each scene in the dataset consists of a textured 3D mesh reconstruction of interiors such as multi-floor residences, stores, and other private indoor spaces. HM3D surpasses existing datasets available for academic research in terms of physical scale, completeness of the reconstruction, and visual fidelity. HM3D contains 112.5k m^2 of navigable space, which is 1.4 - 3.7x larger than other building-scale datasets such as MP3D and Gibson. When compared to existing photorealistic 3D datasets such as Replica, MP3D, Gibson, and ScanNet, images rendered from HM3D have 20 - 85% higher visual fidelity w.r.t. counterpart images captured with real cameras, and HM3D meshes have 34 - 91% fewer artifacts due to incomplete surface reconstruction. The increased scale, fidelity, and diversity of HM3D directly impacts the performance of embodied AI agents trained using it. In fact, we find that HM3D is `pareto optimal' in the following sense -- agents trained to perform PointGoal navigation on HM3D achieve the highest performance regardless of whether they are evaluated on HM3D, Gibson, or MP3D. No similar claim can be made about training on other datasets. HM3D-trained PointNav agents achieve 100% performance on Gibson-test dataset, suggesting that it might be time to retire that episode dataset.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings (2022, methodology): “Experimental Setup We training our SemanticNav agents using the 800 training environments 112 from HM3D [19], and measure performance on one ImageNav and three ObjectNav datasets.”
OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, neutral): “To establish benchmarks for EM-EQA and A-EQA, we collect a human-generated dataset of ( Q, H, A ⇤ ) using videos [6] and 3D scans of real-world environments [32, 36, 39, 45].”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, neutral): “…of publicly available datasets, including Kinetics-710 (Kay et al., 2017), SthSthv2 (Goyal et al., 2017), NExtQA (Xiao et al., 2021), CLEVRER (Yi et al., 2019), EgoQA (Fan, 2019), Tgif (Li et al., 2016), WebVidQA(Yang et al., 2021), RealworldQA (X. AI, 2024), and Hm3d (Ramakrishnan et al., 2021).”
Objaverse-XL: A Universe of 10M+ 3D Objects (2023, background): “…ScanNet3D [31], Articulated Knowledge Base-48 (AKB-48) [32] and OmniObjects3D [23] for objects, and Matterport3D [33] and Habitat-Matterport 3D [34] for scenes, improve on the diversity or quality of their 3D models, by either scanning objects or using more advanced modeling techniques, but…”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, methodology): “In comparison, the Embodied AI (E-AI) research community predominantly trains agents in simulators with far fewer scenes [94, 63, 27].”
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? (2023, methodology): “The dataset is comprised of 80 training, 20 validation, and 20 testing scenes and utilizes the Habitat simulator [15, 65] and HM3D [68] environments.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “This method has shown broad application potential in the ﬁeld of embodied AI.”
VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation (2023, methodology): “Specifically, we achieve absolute increases in success rates weighted by path length over prior state-of-the-art approaches of 12% on Gibson [6], 5% on Matterport 3D (MP3D) [7] and 3% on Habitat-Matterport 3D (HM3D) [8] datasets.”
An Embodied Generalist Agent in 3D World (2023, neutral): “Ad-6 ditionally, we test generalization to the validation split of the newly introduced HM3D ObjNav task (Ramakrishnan et al., 2021).”
Simple but Effective: CLIP Embeddings for Embodied AI (2021, neutral): “Large scene datasets accompany these simulators [4, 24, 37].”
Learning Interactive Real-World Simulators (2023, methodology): “The history conditioned base model operates at temporal and spatial resolution [16 , 24 , 40] , and the two spatial super-resolution models operate at spatial resolution [24 , 40] → [48 , 80] and [48 , 80] → [192 , 320] , respectively.”
3D-LLM: Injecting the 3D World into Large Language Models (2023, background): “• Habitat-Matterport (HM3D) [39] is a dataset of 3D environments of embodied AI.”
Knowledge Enhanced Efficient Robotic Decision Making with Intuitive-Deliberate LLMs (2025, unknown): “Experimental evaluation on the Habitat simulator with the HM3D dataset [19] demonstrates our method’s superior success rate among compared approaches while maintaining competitive path efficiency.”