Learning Depth with Convolutional Spatial Propagation Network

By Xinjing Cheng, Peng Wang, Ruigang Yang. 2018

378 citations recorded by the source corpus.

In this paper, we propose the convolutional spatial propagation network (CSPN) and demonstrate its effectiveness for various depth estimation tasks. CSPN is a simple and efficient linear propagation model, where the propagation is performed with a manner of recurrent convolutional operations, in which the affinity among neighboring pixels is learned through a deep convolutional neural network (CNN). Compare to the previous state-of-the-art (SOTA) linear propagation model, i.e., spatial propagation networks (SPN), CSPN is <inline-formula><tex-math notation="LaTeX">$\mathbf 2$</tex-math><alternatives><mml:math><mml:mn mathvariant="bold">2</mml:mn></mml:math><inline-graphic xlink:href="cheng-ieq1-2947374.gif"/></alternatives></inline-formula> to <inline-formula><tex-math notation="LaTeX">$\mathbf 5\times$</tex-math><alternatives><mml:math><mml:mrow><mml:mn mathvariant="bold">5</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="cheng-ieq2-2947374.gif"/></alternatives></inline-formula> faster in practice. We concatenate CSPN and its variants to SOTA depth estimation networks, which significantly improve the depth accuracy. Specifically, we apply CSPN to two depth estimation problems: depth completion and stereo matching, in which we design modules which adapts the original 2D CSPN to embed sparse depth samples during the propagation, operate with 3D convolution and be synergistic with spatial pyramid pooling. In our experiments, we show that all these modules contribute to the final performance. For the task of depth completion, our method reduce the depth error over 30 percent in the NYU v2 and KITTI datasets. For the task of stereo matching, our method currently ranks <inline-formula><tex-math notation="LaTeX">$\mathbf 1st$</tex-math><alternatives><mml:math><mml:mrow><mml:mn mathvariant="bold">1</mml:mn><mml:mi>s</mml:mi><mml:mi>t</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="cheng-ieq3-2947374.gif"/></alternatives></inline-formula> on both the KITTI Stereo 2012 and 2015 benchmarks.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, background): “therefore necessary, which is in fact possible through single camera [156] though stereo or multi-view systems are more robust [157].”
Self-Supervised Monocular Depth Hints (2019, methodology): “[3, 2, 29], however most modern stereo algorithms are supervised using the LiDAR ground truth from the KITTI dataset.”
DeepVCP: An End-to-End Deep Neural Network for Point Cloud Registration (2019, background): “important category of the problem, has seen many recent developments with emerging results in solving vision problems, including stereo matching [47, 5], depth estimation [36] and SFM [40, 51].”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, background): “Recent approaches improved the underlying architectures and contributed more effective ways of regularizing the cost volume [7, 94, 95, 10].”
Dense Depth Priors for Neural Radiance Fields from Sparse Input Views (2021, methodology): “Instead, we leverage recent work on 2D sparse depth completion that directly completes depth maps in image space [3, 4] and extend it to also predict uncertainty.”
DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch (2019, methodology): “GC-Net [15] SegStereo [35] CRL [25] PDS-Net [33] PSM-Net [8] CSPN [9] Our-Best DispNetC [23] Our-Fast EPE 2.”
Learning Guided Convolutional Network for Depth Completion (2019, neutral): “Totally, we evaluate our model on 3944 image frames, with ‘DDP’ [55], ‘DeepLiDAR’ [6], ‘CSPN’ [35], [56] and ‘NConv-CNN’ [33].”
CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching (2021, background): “Actually, a majority of the top-performing networks in public leaderboards belong to this category, such as GANet [36] and CSPN [5].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, positive): “In this work, the authors used COLMAP to extract a sparse point cloud, which was processed by a Depth Completion Network [86] to produce depth and uncertainty maps.”
Artificial intelligence in the creative industries: a review (2020, methodology): “st volume network (SCV-Net) [288], a fast densenet [199] and a guided aggregation net (GA-Net) [198]. On the KITTI Stereo dataset benchmark [289], the convolutional spatial propagation network (CSPN) [204] ranks 1st at the time of writing (the number of erroneous pixels reported as 1.74%). CSPN has ”