Hierarchical Deep Stereo Matching on High-Resolution Images

By Gengshan Yang, Joshua M. Manela, Michael Happold, Deva Ramanan. Computer Vision and Pattern Recognition, 2019

296 citations recorded by the source corpus.

We explore the problem of real-time stereo matching on high-res imagery. Many state-of-the-art (SOTA) methods struggle to process high-res imagery because of memory constraints or speed limitations. To address this issue, we propose an end-to-end framework that searches for correspondences incrementally over a coarse-to-fine hierarchy. Because high-res stereo datasets are relatively rare, we introduce a dataset with high-res stereo pairs for both training and evaluation. Our approach achieved SOTA performance on Middlebury-v3 and KITTI-15 while running significantly faster than its competitors. The hierarchical design also naturally allows for anytime on-demand reports of disparity by capping intermediate coarse results, allowing us to accurately predict disparity for near-range structures with low latency (30ms). We demonstrate that the performance-vs-speed tradeoff afforded by on-demand hierarchies may address sensing needs for time-critical applications such as autonomous driving.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/cdbef23475c744f0e8ebb62dd9ced57948e2442c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching (2021, methodology): “Asymmetric chromatic augmentation and asymmetric occlusion [33] are employed for data augmentation.”
Simple Multi-dataset Detection (2021, background): “In recent years, training on multiple diverse datasets has emerged as an effective tool to improve model robustness for depth estimation [29], stereo matching [43], and person detection [13].”
Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers (2020, background): “Following the same idea of computing matching cost by learnt 3D convolutions, other works [40, 7, 41, 15] attempt to enable high resolution inference, mitigate the memory constraint, and/or leverage richer context information via a multi-resolution approach.”
RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching (2021, methodology): “Specialized approaches are required to operate at high resolutions [42] such as the mega-pixel images from the Middlebury dataset [28].”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, methodology): “(5) HighResNet [32]: To reﬁne both the spatial and the depth resolutions while operating on high resolution images, this method searches for correspondences incrementally using a coarse-to-ﬁne hierarchy.”
HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching (2020, methodology): “Recent methods such as GA-Net [61] and HSM [54] are obtaining slightly better metrics, although they require 1 .”
RAFT: Recurrent All-Pairs Field Transforms for Optical Flow (2020, methodology): “Following HSM-Net [43], we also randomly erase rectangular regions in I2 with probability 0.”
Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (2019, background): “HSM [40] builds a light model for high-resolution images with a hierarchical design.”
Hierarchical Neural Architecture Search for Deep Stereo Matching (2020, background): “GA-Net [33] proposes a semi-global aggregation layer and a local guided aggregation layer to capture the local and the whole-image cost dependencies respectively.”
Unifying Flow, Stereo and Depth Estimation (2022, methodology): “More specifically, we first finetune the Scene Flow pretrained model on the mixed Scene Flow [58], Tartan Air [109], Falling Things [110], CARLA HR-VS [107], CREStereo Dataset [62], InStereo2K [111] and Middlebury [15], [112] datasets for 100K iterations.”
Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation (2022, background): “Another line of networks [7, 17, 18, 51, 55] uses 3D convolutions to perform cost volume construction and aggregation in traditional methods.”
Attention Concatenation Volume for Accurate and Efficient Stereo Matching (2022, background): “Recently, CNN-based stereo models [7, 9, 12, 14, 21, 23] have achieved impressive performance on most of the standard benchmarks.”