Cascade Residual Learning: A Two-Stage Convolutional Neural Network for Stereo Matching

By Jiahao Pang, Wenxiu Sun, Jimmy S. J. Ren, Chengxi Yang, Qiong Yan. 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), 2017

483 citations recorded by the source corpus.

Leveraging on the recent developments in convolutional neural networks (CNNs), matching dense correspondence from a stereo pair has been cast as a learning problem, with performance exceeding traditional approaches. However, it remains challenging to generate high-quality disparities for the inherently ill-posed regions. To tackle this problem, we propose a novel cascade CNN architecture composing of two stages. The first stage advances the recently proposed DispNet by equipping it with extra up-convolution modules, leading to disparity images with more details. The second stage explicitly rectifies the disparity initialized by the first stage; it couples with the first-stage and generates residual signals across multiple scales. The summation of the outputs from the two stages gives the final disparity. As opposed to directly learning the disparity at the second stage, we show that residual learning provides more effective refinement. Moreover, it also benefits the training of the overall cascade network. Experimentation shows that our cascade residual learning scheme provides state-of-the-art performance for matching stereo correspondence. By the time of the submission of this paper, our method ranks first in the KITTI 2015 stereo benchmark, surpassing the prior works by a noteworthy margin.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/439f205642eaebafa95047be9933a3872aa66553

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Pyramid Stereo Matching Network (2018, methodology): “We also compared the performance of PSMNet with other state-of-the-art methods, including CRL [21], Disp-NetC [19], GC-Net [13], using the Scene Flow test set.”
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (2019, background): “Re lated approaches have also been proposed in the stereo liter ature [18, 34, 44], successfully improving the accuracy but still increasing the network size.”
HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching (2020, methodology): “All these methods rely on expensive cost-volume filtering operations using 3D convolutions [53] or multiple refinement layers [39], preventing real-time performance.”
StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction (2018, methodology): “State of the art [42] uses a hour-glass like architecture in their refinement step, that has been shown to be really effective for inpainting purposes [44].”
Real-Time Self-Adaptive Deep Stereo (2018, methodology): “Thereby, [21] ignited the spread of end-to-end stereo architectures [17, 24, 19, 4, 16, 11] that quickly outmatched any other technique on the KITTI benchmarks by leveraging on a peculiar training protocol.”
Hierarchical Deep Stereo Matching on High-Resolution Images (2019, methodology): “Some recent methods [5, 12, 13, 16, 20, 23] replace post-processing with 2D/3D convolutions applied to the cost volume, producing SOTA performance on the KITTI benchmark.”
Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation (2018, methodology): “Pang et al. [31] extended DispNetC by stacking a reﬁnement network on top, similar to FlowNet 2.0 [18], with the diﬀerence that the second network is posed in a residual setting.”
Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry (2018, methodology): “Similar residual learning architectures have been successfully applied to related deep learning tasks [20,34].”
Learning for Disparity Estimation Through Feature Constancy (2017, unknown): “To further demonstrate the generalization capability of our method, the iResNet-i2, CRL [19] and DispNetC [17] models are first trained on the KITTI 2015 training set and then tested on the KITTI 2015 and 2012 test sets without fine-tuning.”
Deep learning in optical metrology: a review (2022, background): “Subpixel refinement: Pang et al.(347) proposed a cascade (two-stage) CNN architecture for subpixel stereomatching.”
DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch (2019, result): “We compare our approach against the best performing algorithms [41, 15, 9, 25, 29, 8] and the real-time models [32, 23].”
Recursive Cascaded Networks for Unsupervised Medical Image Registration (2019, background): “They assign different tasks and inputs to each cascade in a non-recursive way and train them one by one [30, 45], but their performance approaches a limit with only a few (no more than 3) cascades.”
Image Matching from Handcrafted to Deep Features: A Survey (2020, methodology): “For instance, Mayer et al. (2016) trained an end-to-end CNN in their DispNet to obtain a fine disparity map, which is extended by Pang et al. (2017) with a two-stage CNN called cascade residual learning (CRL).”
BA-Net: Dense Bundle Adjustment Network (2018, methodology): “It also starts to surpass most conventional methods in some middle-level tasks such as optical flow [14, 27], stereo matching [28, 39], and monocular depth estimation [16, 32, 53].”
DSNet: Joint Learning for Scene Segmentation and Disparity Estimation (2019, methodology): “Although numerous CNN models have been proposed for scene parsing [24] [4] [43] [3] and disparity estimation [26] [28] [19] [2], it is still less explored to join these tasks in a single deep network Typical deep multi-task learning approaches mainly focused on the ﬁnal prediction level via…”