StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction

By Sameh Khamis, Sean Ryan Fanello, Christoph Rhemann, Adarsh Kowdle, Julien P. C. Valentin, Shahram Izadi. European Conference on Computer Vision, 2018

397 citations recorded by the source corpus.

This paper presents StereoNet, the first end-to-end deep architecture for real-time stereo matching that runs at 60fps on an NVidia Titan X, producing high-quality, edge-preserved, quantization-free disparity maps. A key insight of this paper is that the network achieves a sub-pixel matching precision than is a magnitude higher than those of traditional stereo matching approaches. This allows us to achieve real-time performance by using a very low resolution cost volume that encodes all the information needed to achieve high disparity precision. Spatial precision is achieved by employing a learned edge-aware upsampling function. Our model uses a Siamese network to extract features from the left and right image. A first estimate of the disparity is computed in a very low resolution cost volume, then hierarchically the model re-introduces high-frequency details through a learned upsampling function that uses compact pixel-to-pixel refinement networks. Leveraging color input as a guide, this function is capable of producing high-quality edge-aware output. We achieve compelling results on multiple benchmarks, showing how the proposed method offers extreme flexibility at an acceptable computational budget.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0c1a6ccfe2b507d6edaa856c1e00b29dc1307d5b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Group-Wise Correlation Stereo Network (2019, background): “There are also other works focusing on real-time stereo matching [7] and application friendly stereo [24].”
Hierarchical Deep Stereo Matching on High-Resolution Images (2019, methodology): “Some recent methods [5, 12, 13, 16, 20, 23] replace post-processing with 2D/3D convolutions applied to the cost volume, producing SOTA performance on the KITTI benchmark.”
Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation (2022, background): “Another line of networks [7, 17, 18, 51, 55] uses 3D convolutions to perform cost volume construction and aggregation in traditional methods.”
Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras (2019, methodology): “Other methods learn directly the stereo disparity [17, 18, 43].”
AANet: Adaptive Aggregation Network for Efficient Stereo Matching (2020, methodology): “, StereoNet [15], which are usually based on a very low-resolution cost volume to achieve fast speed, while at the cost of sacrificing accuracy.”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, methodology): “This process can also be run progressively by cascading many of such refinement blocks, each block refines the estimate of the previous block [62], [72].”
Hierarchical Neural Architecture Search for Deep Stereo Matching (2020, background): “Different designs of the feature net and the matching net form variants of the volumetric networks [4, 10, 11, 12, 6].”
Real-Time Self-Adaptive Deep Stereo (2018, methodology): “GWCNet [11] DispNetC [21] StereoNet [18] MADNet”
DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch (2019, background): “They achieve state-of-the-art results by learning deep representations [22, 23] and building deep cost volumes [15, 8, 16].”