MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects

By Martin Rünz, Lourdes de Agapito. International Symposium on Mixed and Augmented Reality, 2018

429 citations recorded by the source corpus.

We present MaskFusion, a real-time, object-aware, semantic and dynamic RGB-D SLAM system that goes beyond traditional systems which output a purely geometric map of a static scene. MaskFusion recognizes, segments and assigns semantic class labels to different objects in the scene, while tracking and reconstructing them even when they move independently from the camera. As an RGB-D camera scans a cluttered scene, image-based instance-level semantic segmentation creates semantic object masks that enable realtime object recognition and the creation of an object-level representation for the world map. Unlike previous recognition-based SLAM systems, MaskFusion does not require known models of the objects it can recognize, and can deal with multiple independent motions. MaskFusion takes full advantage of using instance-level semantic segmentation to enable semantic labels to be fused into an object-aware map, unlike recent semantics enabled SLAM systems that perform voxel-level semantic segmentation. We show augmented-reality applications that demonstrate the unique features of the map output by MaskFusion: instance-aware, semantic and dynamic. Code will be made available.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/5b330e085088b5de0c9aa134e55d9e07773d2ab3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals (2019, methodology): “We obtained the reported results by using the open source implementations available for the different approaches, with the exception of MaskFusion, where we only report results from the paper [13].”
Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping (2019, neutral): “[35], Fusion++ [36], Mask-fusion [29], Cofusion [37], and MID-Fusion [38].”
3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans (2020, background): “Approaches that deal with moving objects, such as DynamicFusion [71], Mask-fusion [91], Co-fusion [90], and MID-Fusion [105] are currently limited to small table-top scenes and focus on objects or dense maps, rather than scene graphs.”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “Most of these works rely on GPU processing (McCormac et al. 2017; Zheng et al. 2019; Tateno et al. 2015; Li et al. 2016; McCormac et al. 2018; Runz et al. 2018; Rünz and Agapito 2017; Xu et al. 2019).”
DynaSLAM II: Tightly-Coupled Multi-Object Tracking and SLAM (2020, methodology): “Since it is of crucial importance for dense approaches to obtain a highly accurate segmentation mask, the works Mask-Fusion [22] and MID-Fusion [23] reﬁne it by assuming that human-made objects are largely convex.”
EM-Fusion: Dynamic Object-Level SLAM With Probabilistic Data Association (2019, methodology): “By exporting the relative trajectory of the teddy bear and the camera, we can compare the object trajectory to the ground truth camera trajectory as was done in [16].”
In-Place Scene Labelling and Understanding with Implicit Scene Representation (2021, methodology): “For example, [6] uses point clouds, [13] and [22] use surfels, [18] uses voxels, and [12] uses signed distance ﬁelds.”
SuMa++: Efficient LiDAR-based Semantic SLAM (2019, methodology): “Motivated by the advances of deep learning and Convolutional Neural Networks (CNNs) for scene understanding, there have been many semantic SLAM techniques exploiting this information using cameras [5], [31], cameras + IMU data [4], stereo cameras [10], [15], [18], [33], [38], or RGB-D sensors [3], [”
Open-vocabulary Queryable Scene Representations for Real World Planning (2022, background): “Many representations are proposed, ranging from a faithful 3D recontruction [12] of the environment, to more object-centric ones [13], [14], such as object detection bounding boxes [15] and 3D bounding boxes [16].”
RigidFusion: RGB‐D Scene Reconstruction with Rigidly‐moving Objects (2021, methodology): “For Airship object in Room4 sequence, both MaskFusion [RBA18] and EMFusion [SS19] fails to associating segments across frames and output a fragmented tra-jctories.”
Visual Language Maps for Robot Navigation (2022, background): “The literature has focused on either densely annotating 3D volumetric maps with 2D semantic segmentation CNNs [16] or object-oriented approaches [17], [18], [19], which build 3D maps around detected objects to enable object-level posegraph optimization.”
Fusion++: Volumetric Object-Level SLAM (2018, methodology): “R¨unz and Agapito [29], as in our method, use Mask R-CNN predictions to detect object instances.”
MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM (2018, methodology): “For each RGB-D frame, we use Mask R-CNN [16] to find semantic instances, followed by geometric edge refinement to solve leaked mask boundary [14].”