SemanticFusion: Dense 3D semantic mapping with convolutional neural networks

By John McCormac, Ankur Handa, Andrew J. Davison, Stefan Leutenegger. 2016

615 citations recorded by the source corpus.

Ever more robust, accurate and detailed mapping using visual sensing has proven to be an enabling factor for mobile robots across a wide variety of applications. For the next level of robot intelligence and intuitive user interaction, maps need to extend beyond geometry and appearance — they need to contain semantics. We address this challenge by combining Convolutional Neural Networks (CNNs) and a state-of-the-art dense Simultaneous Localization and Mapping (SLAM) system, ElasticFusion, which provides long-term dense correspondences between frames of indoor RGB-D video even during loopy scanning trajectories. These correspondences allow the CNN's semantic predictions from multiple view points to be probabilistically fused into a map. This not only produces a useful semantic 3D map, but we also show on the NYUv2 dataset that fusing multiple predictions leads to an improvement even in the 2D semantic labelling over baseline single frame predictions. We also show that for a smaller reconstruction dataset with larger variation in prediction viewpoint, the improvement over single frame segmentation increases. Our system is efficient enough to allow real-time interactive use at frame-rates of ≈25Hz.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes (2017, methodology): “With ScanNet, we close this gap as we have captured a sufﬁciently large amount of 3D data to use real-world RGB-D input for both training and test sets.”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “McCormac et al. (2016) propose a pipeline for dense 3D semantic mapping designed to work online by fusing semantic predictions of a CNN with the geometric information from a SLAM system (ElasticFusion by Whelan et al.”
SceneNet RGB-D: Can 5M Synthetic Images Beat Generic ImageNet Pre-training on Indoor Segmentation? (2017, methodology): “Firstly, our dataset explicitly provides a randomly generated sequential video trajectory within a scene, allowing 3D correspondences between viewpoints for 3D scene understanding tasks, with the ground truth camera poses acting in lieu of a SLAM system [21].”
Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations (2018, methodology): “In particular, we consider SemanticFusion [1], where the SLAM reconstruction is carried out by ElasticFusion [53], which relies on RGB-D inputs in order to find dense correspondences between frames.”
Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization (2022, neutral): “The literature has focused on both object-based maps [10, 14, 41, 42, 53, 56] and dense maps, including volumetric models [23, 38, 40], point clouds [6, 32, 61], and 3D meshes [48, 51].”
In-Place Scene Labelling and Understanding with Implicit Scene Representation (2021, methodology): “In semantic mapping systems (e.g. [29, 13, 18]), multiple 2D semantic observations are integrated into a 3D map or target frames to produce a more consistent and accurate semantic segmentation.”
Fusion++: Volumetric Object-Level SLAM (2018, background): “We found that the standard multiplicative Bayesian update scheme [15, 20]:”
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, methodology): “Aside from reconstructing 3D geometry, recent works also use deep learning-based object detection and segmentation models to reconstruct the 3D scene representations with dense semantic mapping [8]–[11] or object-level decomposition [12]–[15].”
QuadricSLAM: Dual Quadrics From Object Detections as Landmarks in Object-Oriented SLAM (2018, background): “SemanticFusion [24] recently demonstrated how a dense 3-D reconstruction obtained by SLAM can be enriched with semantic information.”
Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems (2021, methodology): “In single-robot SLAM, metric-semantic models have been employed in pioneering work such as SLAM++ [3] and SemanticFusion [4].”
Global Pose Estimation with an Attention-Based Recurrent Network (2018, methodology): “SemanticFusion [29] combines convolutional neural networks (CNNs) and a dense ElasticFusion [47].”
ImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes (2020, background): “Semantic segmentation of 3D scenes often uses both RGB and depth data [10, 27].”
SuMa++: Efficient LiDAR-based Semantic SLAM (2019, methodology): “Motivated by the advances of deep learning and Convolutional Neural Networks (CNNs) for scene understanding, there have been many semantic SLAM techniques exploiting this information using cameras [5], [31], cameras + IMU data [4], stereo cameras [10], [15], [18], [33], [38], or RGB-D sensors [3], [”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, methodology): “Broadly speaking, there are two approaches to solving this problem: 1) Predict semantics on 2D input images using a 2D segmentation network [1,7,22] and back project the labels to 3D [35–37] 2) Directly predict the semantic labels in the 3D space.”
Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges (2017, methodology): “In [41] the method fused a map built using elastic fusion [42] and semantic segmentation from convolutional neural networks termed as semantic fusion.”
JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds With Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields (2019, background): “, random forests [14, 36, 46, 42], or deep neural networks [27, 49, 30].”
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018, neutral): “l wind. avg. SceneNet [36] 70.8 5.5 76.2 59.6 95.9 62.3 50.0 18.0 61.3 42.2 22.2 86.1 32.1 52.5 Hermans et al. [37] 68.4 45.4 83.4 41.9 91.5 37.1 8.6 35.8 58.5 27.7 38.4 71.8 48.0 54.3 SemanticFusion [38] (RGBD+CRF) 62.0 58.4 43.3 59.5 92.7 64.4 58.3 65.8 48.7 34.3 34.3 86.3 62.3 59.2 SemanticFusion”
FutureMapping: The Computational Structure of Spatial AI Systems (2018, methodology): “This ‘SLAM’ element may either be embedded within a specially architected neural network, or be more explicitly separated from machine learning in another module which stores and updates map properties using standard estimation methods (as in SemanticFusion [33] for instance).”
Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping (2019, methodology): “Examples include SemanticFusion [17], the approach of Zheng et al.”
OpenScene: 3D Scene Understanding with Open Vocabularies (2022, background): “Another line of research performs 3D scene understanding experiments with only 2D ground truth supervisions [16, 27, 38, 47, 53, 57].”