Matterport3D: Learning from RGB-D Data in Indoor Environments
By Angel X. Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, Yinda Zhang. 2017
2,583 citations recorded by the source corpus.
Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms. However, existing datasets still cover only a limited number of views or a restricted scale of spaces. In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. Annotations are provided with surface reconstructions, camera poses, and 2D and 3D semantic segmentations. The precise global alignment and comprehensive, diverse panoramic set of views over entire buildings enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification.
Subfields: Computer Vision and Pattern Recognition, Environmental Engineering, Geology, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Jitendra Malik
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Andrew Zisserman
- Cordelia Schmid
In Environmental Engineering
- Matthias Nießner
- Jitendra Malik
- Angela Dai
- Silvio Savarese
- Saurabh Gupta
- Thomas A. Funkhouser
- Dhruv Batra
- Kristen Grauman
In Geology
- Matthias Nießner
- Jitendra Malik
- Angela Dai
- Andreas Geiger
- Leonidas J. Guibas
- Or Litany
- Silvio Savarese
- Charles Ruizhongtai Qi
In Representation & Self-Supervised Learning
- Trevor Darrell
- Andrea Vedaldi
- Andrew Zisserman
- Kristen Grauman
- Antonio Torralba
- Abhinav Kumar Gupta
- Jitendra Malik
- Dacheng Tao
Citations in the corpus
- How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “R2R is built on the indoor environments from the MatterPort3D dataset (Chang et al., 2017).”
- ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, neutral): “These benchmarks include language-driven navigation and embodied question answering, which have seen dramatic improvements in modeling thanks to environments like Matterport 3D [3, 11], AI2-THOR [25], and AI Habitat [44].”
- Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, methodology): “We construct our pre-training dataset based on the Matterport3D Simulator, a photo-realistic visual reinforcement learning (RL) simulation environment for the development of intelligent agents based on the Matterport3D dataset [5].”
- Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, methodology): “In contrast to these datasets, the recently released Matterport3D dataset [9] contains a comprehensive set of panoramic views.”
- Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “…the color image. alistic synthetic datasets and interactive simulation environments that have spurred rapid progress towards the goal of holistic indoor scene understanding [6, 7, 9, 10, 14, 15, 19, 22, 24, 31, 33, 36, 37, 39, 43, 44, 45, 46, 49, 52, 61, 62, 63, 65, 70, 72, 75, 76, 79, 83, 84].”
- A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “3) Datasets: As in visual exploration, Matterport3D and Gibson V1 are the most popular datasets.”
- The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation (2018, methodology): “The Stanford 2D-3D-S [13], ScanNet [14] and Matterport3D [12] datasets provide 3D real-world scenes with raw 3D point clouds and reconstructed meshes.”
- Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation (2018, methodology): “For indoor navigation simulators like AI2-THOR [29], SUNCG [30] and Matterport3D [31] have emerged.”
- AI2-THOR: An Interactive 3D Environment for Visual AI (2017, background): “HoME [5], House3D [2] and MINOS [17] (which are based on SUNCG [18] and/or Matterport3D [6]) and SceneNet RGBD [15] also provide synthetic indoor environments.”
- Taskonomy: Disentangling Task Transfer Learning (2018, methodology): “The images are registered on and aligned with buildingwide meshes similar to [3, 101, 14] enabling us to programmatically compute the ground truth for many tasks without human labeling.”
- Cognitive Mapping and Planning for Visual Navigation (2017, methodology): “Numerous works such as (Chang et al. 2017; Dai et al. 2017; Ammirato et al. 2017) have collected large-scale datasets consisting of scans of real world environments, while (Savva et al. 2017; Wu et al. 2018; Xia et al. 2018) have built more sophisticated simulation environments based on such scans.”
- Speaker-Follower Models for Vision-and-Language Navigation (2018, methodology): “The dataset consists of 7,189 paths sampled from the Matterport3D Evaluation metrics.”
- Habitat: A Platform for Embodied AI Research (2019, methodology): “The availability of large-scale 3D scene datasets [5, 24, 8] and community interest in active vision tasks led to a recent surge of work that resulted in the development of a variety of simulation platforms for indoor environments [17, 7, 13, 22, 26, 3, 27, 28, 21].”
- Learning to Speak and Act in a Fantasy Text Adventure Game (2019, methodology): “A number of visual, rather than text, platforms have been proposed such as House3D (Wu et al., 2018b), HoME (Brodeur et al., 2017), MINOS (Savva et al., 2017), Matterport3D (Chang et al., 2017) and AI2-THOR (Kolve et al., 2017), and the Minecraft MALMO project (Johnson et al., 2016), but they typica”
- Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, background): “Alternative options, yet smaller in size, include Al2-THOR (Zhu et al., 2017), SceneNet RGB-D (McCormac et al., 2017), Stanford 3D (Armeni et al., 2016) and Matterport 3D (Chang et al., 2017).”
- Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, methodology): “R2R Dataset We evaluate our approaches on the Roomto-Room (R2R) dataset [2] for vision-language navigation in real 3D environments, which is built upon the Matterport3D dataset [5].”
- MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments (2017, methodology): “At the time of writing, the simulator provides immediate support for two datasets: the SUNCG dataset of synthetic furnished houses (Song et al., 2017) and the Matterport3D dataset of reconstructed real buildings (Chang et al., 2017).”
- Vision-and-Dialog Navigation (2019, methodology): “We collect 2050 human-human navigation dialogs, comprising over 7k navigation trajectories punctuated by question-answer exchanges, across 83 MatterPort [6] houses.1 We prompt with initial instructions that are both ambiguous and underspecified.”
- On Evaluation of Embodied Navigation Agents (2018, methodology): “We adopt the 61 / 11 / 18 training/validation/test house splits specified by the original dataset [7].”
- Improving Vision-and-Language Navigation with Image-Text Pairs from the Web (2020, methodology): “Images from the Conceptual Captions (CC) [24] (top) and Matterport3D (MP3D) [5] (bottom) datasets illustrate the differences between the two domains.”