Learning for Disparity Estimation Through Feature Constancy

By Zhengfa Liang, Yiliu Feng, Yulan Guo, Hengzhu Liu, Linbo Qiao, Li Zhou, Jianfeng Zhang. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

353 citations recorded by the source corpus.

Stereo matching algorithms usually consist of four steps, including matching cost calculation, matching cost aggregation, disparity calculation, and disparity refinement. Existing CNN-based methods only adopt CNN to solve parts of the four steps, or use different networks to deal with different steps, making them difficult to obtain the overall optimal solution. In this paper, we propose a network architecture to incorporate all steps of stereo matching. The network consists of three parts. The first part calculates the multi-scale shared features. The second part performs matching cost calculation, matching cost aggregation and disparity calculation to estimate the initial disparity using shared features. The initial disparity and the shared features are used to calculate the feature constancy that measures correctness of the correspondence between two input images. The initial disparity and the feature constancy are then fed into a sub-network to refine the initial disparity. The proposed method has been evaluated on the Scene Flow and KITTI datasets. It achieves the state-of-the-art performance on the KITTI 2012 and KITTI 2015 benchmarks while maintaining a very fast running time. Source code is available at http://github.com/leonzfa/iResNet.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/33389b1da844f13aba317e8b16c814888c26c827

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

RAFT: Recurrent All-Pairs Field Transforms for Optical Flow (2020, methodology): “se any upsampling operations during inference. Iterative Renement for Optical Flow Many recent works have used iterative renement to improve results on optical ow [23,34,37,20,44] and related tasks [25,47,39]. Ilg et al. [23] applied iterative renement to optical ow by stacking multiple FlowNetS and”
Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (2019, neutral): “Methods All (%) Noc (%) D1-bg D1-fg D1-all D1-bg D1-fg D1-all DispNetC [30] 4.32 4.41 4.34 4.11 3.72 4.05 GC-Net [22] 2.21 6.16 2.87 2.02 5.58 2.61 CRL [34] 2.48 3.59 2.67 2.32 3.12 2.45 iResNet-i2e2 [27] 2.14 3.45 2.36 1.94 3.20 2.15 SegStereo [49] 1.88 4.07 2.25 1.76 3.70 2.08 PSMNet [3] 1.86 4.62”
HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching (2020, methodology): “To speed up computation, cascaded models [39, 7, 32, 16, 53, 59] have been proposed to search in disparity space in a coarseto-fine fashion.”
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (2019, background): “Re lated approaches have also been proposed in the stereo liter ature [18, 34, 44], successfully improving the accuracy but still increasing the network size.”
Real-Time Self-Adaptive Deep Stereo (2018, methodology): “Thereby, [21] ignited the spread of end-to-end stereo architectures [17, 24, 19, 4, 16, 11] that quickly outmatched any other technique on the KITTI benchmarks by leveraging on a peculiar training protocol.”
Learning Depth with Convolutional Spatial Propagation Network (2018, methodology): “the Scene Flow test set, we also submitted our results to KITTI 2012 and 2015 test evaluation server to compare against other SOTA methods proposed in recent years, including PSM-Net [26], iResNet-i2 [102], GC-Net [29], EdgeStereo [101], SGMNet [99], Displets v2 [83] and MC-CNN [28].For KITTI benchm”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, result): “While early deep stereo methods used mostly convolutional layers to map from images pairs to depth using ground-truth supervision e.g. [61, 77, 55], [45] showed that integrating a plane-sweep stereo cost volume signiﬁcantly improved results.”
SegStereo: Exploiting Semantic Information for Disparity Estimation (2018, methodology): “[28] integrated disparity estimation and refinement into one network.”
DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch (2019, neutral): “38 3 iResNet ROB [18] 3 2.”
AutoDispNet: Improving Disparity Estimation With AutoML (2019, background): “For instance, state-of-the art disparity estimation methods [44, 37, 68] are fully convolutional and do not contain any pooling operations.”
Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers (2020, background): “Subsequent approaches [25, 22] take advantage of learning-based feature extractors and compute similarities between the feature descriptors for each pixel.”
Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation (2022, background): “One line of networks [12, 21, 22, 27, 29, 44, 49] only uses 2D convolutions.”
AANet: Adaptive Aggregation Network for Efficient Stereo Matching (2020, methodology): “9M), and our AANet performs much better than iResNet-i2[18] on the more challenging KITTI 2015 dataset, demonstrating that our method achieves a better balance between accuracy and speed.”
RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching (2021, background): “[21] first proposed a 2 stage refinement network for stereo.”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, methodology): “The only exception is iResNet [63] trained on Kitti2015 and on ROB [152], which had more than 90% , respectively 55% , of pixels with an error that is larger than 5 pixels.”