Fusion++: Volumetric Object-Level SLAM

By John McCormac, R. Clark, Michael Bloesch, Andrew J. Davison, Stefan Leutenegger. 2018

342 citations recorded by the source corpus.

We propose an online object-level SLAM system which builds a persistent and accurate 3D graph map of arbitrary reconstructed objects. As an RGB-D camera browses a cluttered indoor scene, Mask-RCNN instance segmentations are used to initialise compact per-object Truncated Signed Distance Function (TSDF) reconstructions with object size-dependent resolutions and a novel 3D foreground mask. Reconstructed objects are stored in an optimisable 6DoF pose graph which is our only persistent map representation. Objects are incrementally refined via depth fusion, and are used for tracking, relocalisation and loop closure detection. Loop closures cause adjustments in the relative pose estimates of object instances, but no intra-object warping. Each object also carries semantic information which is refined over time and an existence probability to account for spurious instance predictions. We demonstrate our approach on a hand-held RGB-D sequence from a cluttered office scene with a large number and variety of object instances, highlighting how the system closes loops and makes good use of existing objects on repeated loops. We quantitatively evaluate the trajectory error of our system against a baseline approach on the RGB-D SLAM benchmark, and qualitatively compare reconstruction quality of discovered objects on the YCB video dataset. Performance evaluation shows our approach is highly memory efficient and runs online at 4-8Hz (excluding relocalisation) despite not being optimised at the software level.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

RLBench: The Robot Learning Benchmark & Learning Environment (2019, neutral): “[58]).”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, methodology): “Broadly speaking, there are two approaches to solving this problem: 1) Predict semantics on 2D input images using a 2D segmentation network [1,7,22] and back project the labels to 3D [35–37] 2) Directly predict the semantic labels in the 3D space.”
SuMa++: Efficient LiDAR-based Semantic SLAM (2019, methodology): “Motivated by the advances of deep learning and Convolutional Neural Networks (CNNs) for scene understanding, there have been many semantic SLAM techniques exploiting this information using cameras [5], [31], cameras + IMU data [4], stereo cameras [10], [15], [18], [33], [38], or RGB-D sensors [3], [”
MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM (2018, methodology): “In terms of map representation, Fusion++ is based on discrete voxel grids, which suffers from scalability issues, while we represent all our object models in memory-efficient octree structures.”
In-Place Scene Labelling and Understanding with Implicit Scene Representation (2021, methodology): “There have been several classical pixel-wise semantic fusion approaches [6, 13, 12] to integrate monocular CNN predictions from multiple viewpoints to reﬁne segmentation.”
A Survey on Active Simultaneous Localization and Mapping: State of the Art and New Frontiers (2022, methodology): “Examples of dense metric-semantic maps include Voxblox++ [95] and Kimera [96] (which build upon an SDF), and Fusion++ [97] and [98] (based on voxel maps).”
Towards Bounding-Box Free Panoptic Segmentation (2020, methodology): “ performance of BBFNet on small objects and to experiment with faster segmentation networks [29] towards the goal of expanding the capabilities of visual Simultaneous Localisation and Mapping (vSLAM) [24] with semantics and individual object instances. 6. Acknowledgment We would like to thank Prof. ”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, methodology): “Most of these works rely on GPU processing (McCormac et al. 2017; Zheng et al. 2019; Tateno et al. 2015; Li et al. 2016; McCormac et al. 2018; Runz et al. 2018; Rünz and Agapito 2017; Xu et al. 2019).”
Scan2CAD: Learning CAD Model Alignment in RGB-D Scans (2018, methodology): “A closely related direction to ours (and a possible application) is recognition of objects as a part of a SLAM method, and using the retrieved objects as part of a global pose graph optimization [34, 27].”
Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping (2019, neutral): “[35], Fusion++ [36], Mask-fusion [29], Cofusion [37], and MID-Fusion [38].”
RigidFusion: RGB‐D Scene Reconstruction with Rigidly‐moving Objects (2021, result): “This observation is also reported in the previous work [SJP ∗ 18].”
What Should I Do Now? Marrying Reinforcement Learning and Symbolic Planning (2019, methodology): “A more sophisticated strategy with more frequent detections such as Fusion++[24] could further improve our method (however Fusion++ requires a depth camera).”
EM-Fusion: Dynamic Object-Level SLAM With Probabilistic Data Association (2019, methodology): “Since Mask R-CNN can deliver false detections, we follow [12] and maintain an existence probability pex (i) = Ex (i)/(Ex (i) + NonEx (i)), where for each frame with a Mask R-CNN segmentation available Ex (i) is incremented if the object is matched to a segment and otherwise NonEx (i) is incremented.”
3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans (2020, neutral): “[55] and Fusion++ [69].”
Open-vocabulary Queryable Scene Representations for Real World Planning (2022, background): “Many representations are proposed, ranging from a faithful 3D recontruction [12] of the environment, to more object-centric ones [13], [14], such as object detection bounding boxes [15] and 3D bounding boxes [16].”
Visual Language Maps for Robot Navigation (2022, background): “The literature has focused on either densely annotating 3D volumetric maps with 2D semantic segmentation CNNs [16] or object-oriented approaches [17], [18], [19], which build 3D maps around detected objects to enable object-level posegraph optimization.”
Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization (2022, neutral): “Some approaches combine objects and dense map models [31, 39, 54, 68].”