MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM

By Binbin Xu, Wenbin Li, Dimos Tzoumanikas, Michael Bloesch, Andrew J. Davison, Stefan Leutenegger. 2018

225 citations recorded by the source corpus.

We propose a new multi-instance dynamic RGBD SLAM system using an object-level octree-based volumetric representation. It can provide robust camera tracking in dynamic environments and at the same time, continuously estimate geometric, semantic, and motion properties for arbitrary objects in the scene. For each incoming frame, we perform instance segmentation to detect objects and refine mask boundaries using geometric and motion information. Meanwhile, we estimate the pose of each existing moving object using an object-oriented tracking method and robustly track the camera pose against the static scene. Based on the estimated camera pose and object poses, we associate segmented masks with existing models and incrementally fuse corresponding colour, depth, semantic, and foreground object probabilities into each object model. In contrast to existing approaches, our system is the first system to generate an object-level dynamic volumetric map from a single RGB-D camera, which can be used directly for robotic tasks. Our method can run at 2-3 Hz on a CPU, excluding the instance segmentation part. We demonstrate its effectiveness by quantitatively and qualitatively testing it on both synthetic and real-world sequences.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Luca Carlone
2. Juan D. Tardós
3. Yun Chang
4. José Neira
5. Antoni Rosinol
6. Nathan Hughes
7. Wolfram Burgard
8. Jingnan Shi

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jitendra Malik
3. Abhinav Kumar Gupta
4. Trevor Darrell
5. Kristen Grauman
6. Dacheng Tao
7. Ming-Hsuan Yang
8. Xiaogang Wang

In Geology

1. Juan D. Tardós
2. Andy Zeng
3. Jörg Stückler
4. Yun Chang
5. Marcus Abate
6. José Neira
7. Luca Carlone
8. Carlos Campos

Citations in the corpus

• Continuous Scene Representations for Embodied AI (2022, background): “Dynamic SLAM approaches [41, 48, 60] have been proposed to handle moving objects in a scene.”
• DynaSLAM II: Tightly-Coupled Multi-Object Tracking and SLAM (2020, methodology): “Since it is of crucial importance for dense approaches to obtain a highly accurate segmentation mask, the works Mask-Fusion [22] and MID-Fusion [23] refine it by assuming that human-made objects are largely convex.”
• Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization (2022, unknown): “Some approaches combine objects and dense map models [31, 39, 54, 68].”
• Visual Language Maps for Robot Navigation (2022, background): “The literature has focused on either densely annotating 3D volumetric maps with 2D semantic segmentation CNNs [16] or object-oriented approaches [17], [18], [19], which build 3D maps around detected objects to enable object-level posegraph optimization.”
• EM-Fusion: Dynamic Object-Level SLAM With Probabilistic Data Association (2019, methodology): “MID-Fusion [27] follows a similar approach, but represents the 3D map in volumetric SDFs using octrees.”
• Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “Most of these works rely on GPU processing (Li et al., 2016; McCormac et al., 2018, 2017; Rünz and Agapito, 2017; Rünz et al., 2018; Tateno et al., 2015; Xu et al., 2019; Zheng et al., 2019).”