Co-fusion: Real-time segmentation, tracking and fusion of multiple objects

By Martin Rünz, Lourdes de Agapito. 2017

241 citations recorded by the source corpus.

In this paper we introduce Co-Fusion, a dense SLAM system that takes a live stream of RGB-D images as input and segments the scene into different objects (using either motion or semantic cues) while simultaneously tracking and reconstructing their 3D shape in real time. We use a multiple model fitting approach where each object can move independently from the background and still be effectively tracked and its shape fused over time using only the information from pixels associated with that object label. Previous attempts to deal with dynamic scenes have typically considered moving regions as outliers, and consequently do not model their shape or track their motion over time. In contrast, we enable the robot to maintain 3D models for each of the segmented objects and to improve them over time through fusion. As a result, our system can enable a robot to maintain a scene description at the object level which has the potential to allow interactions with its working environment; even in the case of dynamic scenes.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “2018), Co-fusion (Rünz and Agapito 2017), and MID-Fusion (Xu et al.”
ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals (2019, methodology): “In contrast to other recent purely geometric approaches [13], [15], we do not represent the model using surfels, but in the form of a truncated sign distance function (TSDF).”
DynaSLAM II: Tightly-Coupled Multi-Object Tracking and SLAM (2020, result): “using pixel-wise instance segmentation, showing impressive results [21]–[23].”
RigidFusion: RGB‐D Scene Reconstruction with Rigidly‐moving Objects (2021, methodology): “Delayed Processing For motion segmentation, a frame-to-frame approach, such as [RA17,SJP ∗ 18], cannot guarantee the observed motion signals are sufﬁcient because an object may move slowly.”
MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects (2018, methodology): “Note that while Co-Fusion [39] could use semantic cues to segment the scene, in that case the system was not real-time – only the non-semantic version of Co-Fusion was real-time capable. non-rigid surface reconstruction and multibody formulations for independently moving rigid objects.”
FutureMapping: The Computational Structure of Spatial AI Systems (2018, background): “…since as they continually model the state of the world in front of the camera, they can detect when something is out of place with respect to this model, and therefore dense SLAM systems are now being developed which are moving beyond static scenes to reconstruct and track dynamic scenes [40, 47].”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, methodology): “We consider 5 versatile datasets: Replica [44], ScanNet [13], TUM RGB-D dataset [45], Co-Fusion dataset [39], as well as a self-captured large apartment with multiple rooms.”
3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans (2020, background): “Approaches that deal with moving objects, such as DynamicFusion [71], Mask-fusion [91], Co-fusion [90], and MID-Fusion [105] are currently limited to small table-top scenes and focus on objects or dense maps, rather than scene graphs.”
Resolving 3D Human Pose Ambiguities With 3D Scene Constraints (2019, background): “More interesting directions would be to extend our method to dynamic scenes [59], human-human interaction and to account for scene and body deformation.”
Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping (2019, unknown): “[35], Fusion++ [36], Mask-fusion [29], Cofusion [37], and MID-Fusion [38].”
EM-Fusion: Dynamic Object-Level SLAM With Probabilistic Data Association (2019, methodology): “AT-RMSEs (in cm) of estimated trajectories for the synthetic sequences from Co-Fusion [15].”
MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM (2018, methodology): “Despite showing some promising results based on deep learning, most methods [12], [14], [13] simply leverage”