CNN-SLAM: Real-Time Dense Monocular SLAM with Learned Depth Prediction

By Keisuke Tateno, Federico Tombari, Iro Laina, Nassir Navab. Computer Vision and Pattern Recognition, 2017

781 citations recorded by the source corpus.

Given the recent advances in depth prediction from Convolutional Neural Networks (CNNs), this paper investigates how predicted depth maps from a deep neural network can be deployed for the goal of accurate and dense monocular reconstruction. We propose a method where CNN-predicted dense depth maps are naturally fused together with depth measurements obtained from direct monocular SLAM, based on a scheme that privileges depth prediction in image locations where monocular SLAM approaches tend to fail, e.g. along low-textured regions, and vice-versa. We demonstrate the use of depth prediction to estimate the absolute scale of the reconstruction, hence overcoming one of the major limitations of monocular SLAM. Finally, we propose a framework to efficiently fuse semantic labels, obtained from a single frame, with dense SLAM, so to yield semantically coherent scene reconstruction from a single view. Evaluation results on two benchmark datasets show the robustness and accuracy of our approach.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2ba0873996bc8295102edfc5c9613a3cb96e3a9f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “A sparser set of contributions addresses other sensing modalities, including monocular cameras (e.g., CNN-SLAM (Tateno et al. 2017), VSO (Lianos et al. 2018), VITAMIN-E (Yokozuka et al. 2019), XIVO (Dong et al. 2017)) and lidar (Behley et al. 2019; Dubé et al. 2018).”
SuMa++: Efficient LiDAR-based Semantic SLAM (2019, methodology): “Motivated by the advances of deep learning and Convolutional Neural Networks (CNNs) for scene understanding, there have been many semantic SLAM techniques exploiting this information using cameras [5], [31], cameras + IMU data [4], stereo cameras [10], [15], [18], [33], [38], or RGB-D sensors [3], [”
D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry (2020, methodology): “Recently, there have been many efforts to address this by leveraging deep neural networks [48, 68, 80, 83].”
Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry (2018, methodology): “Some recent monocular VO approaches have integrated monocular depth estimation [46,39] to recover the metric scale by scale-matching.”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, methodology): “It is worth noting that there are also several non-deep-learning methods that also aim to produce consistent sequential depth estimates e.g. [96, 44], in addition to conventional SLAM based methods [65, 63, 16, 89], and SLAM methods that integrate a monocular depth estimation network [74, 4, 52].”
The ApolloScape Open Dataset for Autonomous Driving and Its Application (2018, methodology): “Most recently, CNN-SLAM [67] replaces traditional 3D reconstruction module with a single image depth network, and adopts a segment network for image parsing.”
BA-Net: Dense Bundle Adjustment Network (2018, methodology): “To deal with these problems, we use the network for monocular image depth estimation as a compact parameterization, rather than using it as an initialization like in [47].”
SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM (2023, background): “These approaches range from directly integrating depth predictions from a neural network into SLAM [38], to learning a variational autoencoder that decodes compact optimizable latent codes to depth maps [1, 52], to approaches that simultaneously learn to predict a depth cost volume and tracking…”
Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation (2019, background): “Also, they have many potential industrial applications such as autonomous driving system [4], simultaneous localization and mapping (SLAM) [52] and socially interactive robotics [12].”
Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM (2023, background): “Several recent works [11, 29, 30, 38] leverage deep learning to improve the accuracy and robustness of traditional SLAM, and even achieve dense reconstruction with monocular SLAM.”
FutureMapping: The Computational Structure of Spatial AI Systems (2018, background): “The level of scene representation that has been possible in real-time SLAM has gradually improved, from sparse features (e.g. [8]) to dense (e.g. [42]) maps and now increasingly semantic labels [33, 54].”
Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View (2017, background): “Current methods for direct regressing raw depth values produce blurred results [34, 20, 7], partly due to the viewpointdependent nature of depth maps and the large value variance of depth values even for nearby pixels on the same 3D plane.”
Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping (2019, background): “, CNNSLAM [41], VSO [42], VITAMIN-E [43], XIVO [32]) and lidar (e.”
Multimodal End-to-End Autonomous Driving (2019, background): “semantic segmentation [17–22], monocular depth estimation [23–26], SLAM and place recognition [27–33], etc.”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, methodology): “Similar approaches have been incorporated into SLAM systems such as COLMAP [42,43] and CNN-SLAM [50].”
DUSt3R: Geometric 3D Vision Made Easy (2023, positive): “Recent works on dense visual SLAM from RGB video stream are able to produce high-quality depth maps and camera trajectories [7, 22, 116, 122, 130, 132], but they inherit the traditional limitations of SLAM, e.g . noisy predictions, drifts and out-liers in the pixel correspondences.”