Sparsity Invariant CNNs

By Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, Andreas Geiger. International Conference on 3D Vision, 2017

967 citations recorded by the source corpus.

In this paper, we consider convolutional neural networks operating on sparse inputs with an application to depth upsampling from sparse laser scan data. First, we show that traditional convolutional networks perform poorly when applied to sparse data even when the location of missing data is provided to the network. To overcome this problem, we propose a simple yet effective sparse convolution layer which explicitly considers the location of missing data during the convolution operation. We demonstrate the benefits of the proposed network architecture in synthetic and real experiments with respect to various baseline approaches. Compared to dense baselines, the proposed sparse convolution network generalizes well to novel datasets and is invariant to the level of sparsity in the data. For our evaluation, we derive a novel dataset from the KITTI benchmark, comprising 93k depth annotated RGB images. Our dataset allows for training and evaluating depth upsampling and depth prediction techniques in challenging real-world settings and will be made available upon publication.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/90c80317fa68784a3fe4fc3136bb188895b09fa4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Depth with Convolutional Spatial Propagation Network (2018, background): “Finally, we also submit our results to the KITTI depth completion challenge(4) and show that our results is better than previous SOTA method [74] at the time”
MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer (2022, methodology): “Results on KITTI using the improved ground truth [48].”
SBNet: Sparse Blocks Network for Fast Inference (2018, background): “For instance, sparse batch normalization (BN) [11, 26] is invariant to input sparsity level, and outperforms regular BN in optical ﬂow tasks.”
PENet: Towards Precise and Efficient Image Guided Depth Completion (2021, methodology): “A sparse depth map has about 5% valid pixels and a ground truth dense depth map have around 16% valid pixels [8].”
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision (2019, methodology): “Current applications include many safety-critical tasks, such as street-scene semantic segmentation [10, 6, 51], 3D object detection [42, 28] and depth completion [45, 32].”
Digging Into Self-Supervised Monocular Depth Estimation (2018, methodology): “Additionally, 93% of the Eigen split test frames have higher quality ground truth depths provided by [59].”
MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision (2024, positive): “We evaluate the accuracy of point map estimation on eight diverse datasets: NYUv2 [35], KITTI [47], ETH3D [45], iBims-1 [29], Sintel [8], Google Scanned Objects (GSO) [15], DDAD [22], and DIODE [48].”
Differentiable SLAM-net: Learning Particle SLAM for Visual Navigation (2021, methodology): “Since the original depth information for KITTI dataset is from sparse lidar, we use the completed depth data from the [58] as ground-truth depth.”
Non-Local Spatial Propagation Network for Depth Completion (2020, methodology): “m No 903.1 (g) Yes 889.5 (h) Abs Sum 886.0 (i) Tanh C 886.4 (j) Tanh Abs Sum No 891.3 (k) Binary 892.9 (l) Weighted 884.8 (m) Yes 884.1 Table 3. Quantitative evaluation on the KITTI DC validation set [30] with various congurations. Please refer to the text for details. boring pixels to show the rele”
Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network (2018, background): “Finally, we also submit our results to the new KITTI depth completion challenge 2 and show that our results is better than previous SOTA method [45].”
Deep Depth Completion of a Single RGB-D Image (2018, methodology): “Though depth inpainting has received a lot of attention over the past two decades [64], it has generally been addressed with hand-tuned methods that fill holes by extrapolating boundary surfaces [46] or with Markovian image synthesis [13].”
Partial Convolution based Padding (2018, methodology): “Inpainting methods like [25, 19] take the hole mask into consideration for reweighting the convolution results.”
Multi-Task Multi-Sensor Fusion for 3D Object Detection (2019, methodology): “For multi-task training, we also leverage the dense depth labels from the KITTI’s depth dataset [28].”
Wide-Context Semantic Image Extrapolation (2019, background): “It directly learns semantic features from small-size input, which is both effective and efficient by avoiding bias in common padding and upsampling procedures [33, 40, 26].”
Learning Guided Convolutional Network for Depth Completion (2019, methodology): “The KITTI depth completion dataset [5] contains 86 , 898 frames for training, 1 , 000 frames for validation, and another 1 , 000 frames for testing.”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, methodology): “Nowadays, many benchmarks [40,56] in depth estimation are dominated by algorithms based on neural networks [14,66].”
BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation (2022, methodology): “For the online KITTI depth prediction, we use the official benchmark split [59], which contains around 72K training data, 6K selected validation data, and 500 test data without the ground truth.”
PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation (2017, positive): “Both of these operations involve lossy data quantization and require special models to handle sparsity in the lidar image [34] or in voxel space [27].”
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, background): “[59]) and the Eigen test split [60] (161 images).”
Real-Time Self-Adaptive Deep Stereo (2018, methodology): “, LIDAR) and noise removal by expensive manual intervention or postprocessing [43].”