End-to-End Learning of Geometry and Context for Deep Stereo Regression

By Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry. 2017

1,541 citations recorded by the source corpus.

We propose a novel deep learning architecture for regressing disparity from a rectified pair of stereo images. We leverage knowledge of the problem’s geometry to form a cost volume using deep feature representations. We learn to incorporate contextual information using 3-D convolutions over this volume. Disparity values are regressed from the cost volume using a proposed differentiable soft argmin operation, which allows us to train our method end-to-end to sub-pixel accuracy without any additional post-processing or regularization. We evaluate our method on the Scene Flow and KITTI datasets and on KITTI we set a new stateof-the-art benchmark, while being significantly faster than competing approaches.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning a Multi-View Stereo Machine (2017, methodology): “A recent work [28] shows convincing results for binocular stereo by using an end-to-end learning approach with binocular geometry constraints.”
Unsupervised Learning of Depth and Ego-Motion from Video (2017, background): “, feature matching [18], pose estimation [26], and stereo [10, 27, 53].”
Pyramid Stereo Matching Network (2018, methodology): “We also compared the performance of PSMNet with other state-of-the-art methods, including CRL [21], Disp-NetC [19], GC-Net [13], using the Scene Flow test set.”
StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction (2018, methodology): “Current state of the art [29], achieves an EPE of 0.”
Fast Disparity Estimation Using Dense Networks (2018, methodology): “The image-based method uses an autoencoder style model that processes stereo images into dense disparity estimates, GC-Net [1] and DispNet [2].”
Attention Concatenation Volume for Accurate and Efficient Stereo Matching (2022, methodology): “Finally, the predicted value is computed by the soft argmin function [9],”
Local light field fusion (2019, neutral): “This includes recent angular superresolution methods [Wu et al. 2017; Yeung et al. 2018] that interpolate dense views within a light field camera’s aperture but cannot handle sparser input view sampling since they do not model scene geometry.”
Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction (2018, background): “Kendell et al.[17] proposed an end-to-end learning framework to predict disparity from a stereo pair.”
GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose (2018, background): “[24] proposed 3D convolutions over cost volumes by deep features to better aggregate stereo information.”
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018, methodology): “Multiview networks have also been used in the context of stereo reconstruction [19, 20], leveraging feature projection into 3D to produce consistent reconstruction.”
Learning Depth with Convolutional Spatial Propagation Network (2018, methodology): “Current SOTAmethods for stereo also rely on the advancement of deep networks [26], [28], [29].”
GONet: A Semi-Supervised Deep Learning Approach For Traversability Estimation (2018, methodology): “• Simese Net: In the spirit of [36], we explored using a siamese network with ResNet 152 features.”
Real-Time Self-Adaptive Deep Stereo (2018, methodology): “Thereby, [21] ignited the spread of end-to-end stereo architectures [17, 24, 19, 4, 16, 11] that quickly outmatched any other technique on the KITTI benchmarks by leveraging on a peculiar training protocol.”
Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation (2019, methodology): “Inspired by [22], we apply a differentiable 3D soft argmax, which takes a weighted sum of the sampled hypotheses as the vertex deformation.”
Neighbourhood Consensus Networks (2018, neutral): “Related are also methods that estimate optical ﬂow or stereo disparity such as [24, 16, 15, 39, 6], or their trainable counterparts [8, 40, 19].”
Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers (2020, background): “Concurrently, networks such as [20, 5] build a 4D feature volume by concatenating features at different disparities and learn to compute/aggregate the matching cost by 3D convolutions.”
Group-Wise Correlation Stereo Network (2019, methodology): “Then, the disparity estimation (cid:101) d is given by the soft argmin function [6], where k and p k denote a possible disparity level and the corresponding probability.”
Point-Based Multi-View Stereo Network (2019, background): “Concurrently, 3D cost volume regularization approaches have been proposed [14, 12, 13], where a 3D cost volume is built either in the camera frustum or the scene.”