BundleFusion

By Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Christian Theobalt. 2016

620 citations recorded by the source corpus.

Real-time, high-quality, 3D scanning of large-scale scenes is key to mixed reality and robotic applications. However, scalability brings challenges of drift in pose estimation, introducing significant errors in the accumulated model. Approaches often require hours of offline processing to globally correct model errors. Recent online methods demonstrate compelling results but suffer from (1) needing minutes to perform online correction, preventing true real-time use; (2) brittle frame-to-frame (or frame-to-model) pose estimation, resulting in many tracking failures; or (3) supporting only unstructured point-based representations, which limit scan quality and applicability. We systematically address these issues with a novel, real-time, end-to-end reconstruction framework. At its core is a robust pose estimation strategy, optimizing per frame for a global set of camera poses by considering the complete history of RGB-D input with an efficient hierarchical approach. We remove the heavy reliance on temporal tracking and continually localize to the globally optimized frames instead. We contribute a parallelizable optimization framework, which employs correspondences based on sparse features and dense geometric and photometric matching. Our approach estimates globally optimized (i.e., bundle adjusted) poses in real time, supports robust tracking with recovery from gross tracking failures (i.e., relocalization), and re-estimates the 3D model in real time to ensure global consistency, all within a single framework. Our approach outperforms state-of-the-art online systems with quality on par to offline methods, but with unprecedented speed and scan completeness. Our framework leads to a comprehensive online scanning solution for large indoor environments, enabling ease of use and high-quality results.1

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis (2016, background): “Since the introduction of commodity range sensors such as the Microsoft Kinect, RGB-D scanning has gained a huge momentum in both offline and real-time contexts [26, 3, 28, 42, 4, 7].”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, background): “Some datasets targeted at SLAM applications [8] provide tracked camera poses covering parts of rooms [30] or estimated camera poses for individual rooms [7], and Armeni et al.”
SuperGlue: Learning Feature Matching With Graph Neural Networks (2019, methodology): “Camera poses in SUN3D are estimated from SIFT-based sparse SfM, while ScanNet leverages RGB-D fusion and optimization [4], resulting in significantly more accurate poses.”
BA-Net: Dense Bundle Adjustment Network (2018, methodology): “To demonstrate the effectiveness of our method, we evaluate on the ScanNet (Dai et al., 2017a) and KITTI (Geiger et al., 2012) dataset.”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, methodology): “At the core of nearly all of these methods is the volumetric fusion technique from Curless and Levoy [28], which has been adapted for real-time incremental capture of surface geometry and color [29, 30, 31, 32, 33].”
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions (2016, background): “The results show that 3DMatch is considerably better than state-of-the-art methods at matching keypoints, and when combined with standard RANSAC, significantly outperforms other algorithms for geometric registration.”
Scan2CAD: Learning CAD Model Alignment in RGB-D Scans (2018, background): “Very recent RGB-D reconstruction frameworks combine efficient scene representations with global pose estimation [5], and can even perform online updates with global loop closures [8].”
Deep Global Registration (2020, methodology): “For pairwise registration, we use the 3DMatch benchmark [46] which consists of 3D point cloud pairs from various real-world scenes with ground truth transformations estimated from RGB-D reconstruction pipelines [16, 10].”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “, 2020] or the slam methods [Dai et al., 2017] can be adopted to merge the multi-view data.”
3DMatch: Learning the Matching of Local 3D Geometry in Range Scans (2016, methodology): “We include pooling to benefit from the response filtering properties of max pooling; but since the dimensions of the initial input volume are small, we only include only one layer of pooling to avoid a substantial loss of information.”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, neutral): “, KinectFusion [28] among other works [5, 14, 18, 33].”
RTAB‐Map as an open‐source lidar and visual simultaneous localization and mapping library for large‐scale and long‐term online operation (2018, background): “(cid:129) ElasticFusion (Whelan, Salas ‐ Moreno, Glocker, Davison, & Leuteneg-ger, 2016), Kintinuous (Whelan et al., 2015), BundleFusion (Dai et al., 2017), and InfiniTAM (Kähler, Prisacariu & Murray, 2016) All these previous visual SLAM approaches assume that the camera is never obstructed or that ”
OccuSeg: Occupancy-Aware 3D Instance Segmentation (2020, background): “The past ten years have witnessed a rapid development of real-time 3D reconstruction technologies [33, 34, 5, 47, 15] with the popularity of commercial RGB-D depth sensors like Kinect, Xtion, etc.”
Deferred neural rendering (2019, background): “…context of 3D reconstruction with commodity RGB-D sensors, researchers have made significant progress, enabling robust tracking [Choi et al. 2015; Dai et al. 2017; Izadi et al. 2011; Newcombe et al. 2011; Whelan et al. 2016] and large-scale 3D scene capture [Chen et al. 2013; Nießner et al.…”