CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM

By Michael Bloesch, Jan Czarnowski, R. Clark, Stefan Leutenegger, Andrew J. Davison. 2018

436 citations recorded by the source corpus.

The representation of geometry in real-time 3D perception systems continues to be a critical research issue. Dense maps capture complete surface shape and can be augmented with semantic labels, but their high dimensionality makes them computationally costly to store and process, and unsuitable for rigorous probabilistic inference. Sparse feature-based representations avoid these problems, but capture only partial scene information and are mainly useful for localisation only. We present a new compact but dense representation of scene geometry which is conditioned on the intensity data from a single image and generated from a code consisting of a small number of parameters. We are inspired by work both on learned depth from images, and auto-encoders. Our approach is suitable for use in a keyframe-based monocular dense SLAM system: While each keyframe with a code can produce a depth map, the code can be optimised efficiently jointly with pose variables and together with the codes of overlapping keyframes to attain global consistency. Conditioning the depth map on the image allows the code to only represent aspects of the local geometry which cannot directly be predicted from the image. We explain how to learn our code representation, and demonstrate its advantageous properties in monocular SLAM.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Vox-Fusion: Dense Tracking and Mapping with Voxel-based Neural Implicit Representation (2022, methodology): “CodeSLAM [2] proposes to use an encoder-decoder structure to embed depth maps as low dimensional codes.”
BA-Net: Dense Bundle Adjustment Network (2018, methodology): “Similar depth parameterization is introduced in a recent work, CodeSLAM (Bloesch et al., 2018).”
In-Place Scene Labelling and Understanding with Implicit Scene Representation (2021, methodology): “CodeSLAM [1] uses a compact and optimisable latent code to represent dense scene geometry in a view-based visual odometry system.”
LaDDer: Latent Data Distribution Modelling with a Generative Prior (2020, neutral): “Thanks to all the preferable properties, VAEs have been widely applied in many computer vision applications, including image synthesis [11, 17, 41], human motion modelling [14, 36] and 3D reconstruction [1, 4, 6, 10, 29].”
D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry (2020, methodology): “Besides pose estimation, CodeSLAM [2] delivers dense reconstruction by jointly optimizing the learned prior of the dense geometry together with camera poses.”
NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields (2022, methodology): “For example, using depth estimation networks from monocular images [3], multiple images, as in multi-view stereo [4], or using end-to-end neural networks [5].”
Neural RGB-D Surface Reconstruction (2021, methodology): “Reconstructing objects and scenes can be done using passive stereo systems that rely on stereo matching from two or multiple color views [29, 62], Structure-from-Motion [63], or SLAM-based [20, 21, 23] methods.”
Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry (2018, background): “CodeSLAM [2] focuses on the challenge of dense 3D reconstruction.”
Consistent video depth estimation (2020, methodology): “Several methods estimate depth by integrating motion estimation and multi-view reconstruction using two frames [Ummenhofer et al. 2017; Wang et al. 2019a] or a varying number of frames [Bloesch et al. 2018; Valentin et al. 2018; Zhou et al. 2018].”
Differentiable SLAM-net: Learning Particle SLAM for Visual Navigation (2021, methodology): “For example, CodeSLAM[7] and SceneCode[63] learn a compact representation of the scene; CNN-SLAM[57] learns a CNN-based depth predictor as the front-end of a monocular SLAM system; BA-net [56] learns the feature metric representation and a damping factor for bundle adjustment.”
DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras (2021, methodology): “DeepFactors[9] is the most complete deep SLAM system, building on the earlier CodeSLAM[1].”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, neutral): “[161]).”
SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation (2018, background): “ical in nature: powerful representations learned by deep neural networks trained on large scale data. CNNs have indeed shown consistent progress towards robust scene depth and 3D reconstruction [14], [17], [18]. State-of-the-art approaches in leveraging both data and structural constraints mostly di”
A Review of Visual-LiDAR Fusion based Simultaneous Localization and Mapping (2020, methodology): “Finally, with the development of deep learning, some SLAM applications emerged to imitate the previous proposed approaches [9,30].”
SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM (2023, background): “…approaches range from directly integrating depth predictions from a neural network into SLAM [38], to learning a variational autoencoder that decodes compact optimizable latent codes to depth maps [1, 52], to approaches that simultaneously learn to predict a depth cost volume and tracking [53].”