Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network

By Xinjing Cheng, Peng Wang, Ruigang Yang. European Conference on Computer Vision, 2018

381 citations recorded by the source corpus.

Depth estimation from a single image is a fundamental problem in computer vision. In this paper, we propose a simple yet effective convolutional spatial propagation network (CSPN) to learn the affinity matrix for depth prediction. Specifically, we adopt an efficient linear propagation model, where the propagation is performed with a manner of recurrent convolutional operation, and the affinity among neighboring pixels is learned through a deep convolutional neural network (CNN). We apply the designed CSPN to two depth estimation tasks given a single image: (1) Refine the depth output from existing state-of-the-art (SOTA) methods; (2) Convert sparse depth samples to a dense depth map by embedding the depth samples within the propagation procedure. The second task is inspired by the availability of LiDAR that provides sparse but accurate depth measurements. We experimented the proposed CSPN over the popular NYU v2 [1] and KITTI [2] datasets, where we show that our proposed approach improves not only quality (e.g., 30% more reduction in depth error), but also speed (e.g., 2 to 5\(\times \) faster) of depth maps than previous SOTA methods. The codes of CSPN are available at: https://github.com/XinJCheng/CSPN.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/fdfb33a9af5f0b2c247c95f5a6eb053bcc6194df

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Learning Guided Convolutional Network for Depth Completion (2019, methodology): “For indoor scene, to be consistent with comparative depth completion methods [6], [7], [33], [35], the evaluation metrics are selected as root mean squared error (RMSE), mean absolute relative error (REL) and δ i which means the percentage of predicted pixels where the relative error is less a…”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, methodology): “D is a measure of distance, which can be the L2, the L1 [61], [62], [99], [121], the smooth L1 [64], or the smooth L1 but approximated using the twoparameter robust function ρ(·) [72], [122].”
PENet: Towards Precise and Efficient Image Guided Depth Completion (2021, methodology): “Depth completion aims to produce a dense depth map by completing a sparse depth map, without [8], [9] or with the guidance of a reference image [10], [2], [11], [5].”
Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation (2024, positive): “Deep iterative reﬁnement for geometry: Iterative reﬁnement enables multi-step coarse-to-ﬁne prediction and beneﬁts a wide range of geometry estimation tasks, such as optical ﬂow estimation [46], [75], [76], depth completion [43], [77], [78], and stereo matching [45], [79], [80].”
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image (2018, methodology): “Qualitative comparison with some competing methods [33, 6] are shown in Fig.”
The ApolloScape Open Dataset for Autonomous Driving and Its Application (2018, background): “It requires solving tasks such as learning to do visual 3D scene reconstruction [5], [6], [7], [8], self-localization [9], [10], semantic parsing [11], [12], semantic instance understand-”
Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation (2019, methodology): “In order to encode long-distance correlations, the PAP utilizes non-local similarities within each task, different from the literatures [39, 5] only considering local neighbor relationships.”
Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera (2018, methodology): “We follow an approach similar to [39] with the official train/test data split.”
Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review (2020, methodology): “[44] fed RGB-D images to a convolutional spatial propagation network (CSPN).”
GA-Net: Guided Aggregation Net for End-To-End Stereo Matching (2019, background): “(7) twice with the same weight matrix, which is similar to [5].”
Non-Local Spatial Propagation Network for Depth Completion (2020, methodology): “not been explored in conventional algorithms [6,19,9].”