Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches

By Jure Zbontar, Yann LeCun. 2015

1,469 citations recorded by the source corpus.

We present a method for extracting depth information from a rectified image pair. Our approach focuses on the first stage of many stereo algorithms: the matching cost computation. We approach the problem by learning a similarity measure on small image patches using a convolutional neural network. Training is carried out in a supervised manner by constructing a binary classification data set with examples of similar and dissimilar pairs of patches. We examine two network architectures for this task: one tuned for speed, the other for accuracy. The output of the convolutional neural network is used to initialize the stereo matching cost. A series of post-processing steps follow: cross-based cost aggregation, semiglobal matching, a left-right consistency check, subpixel enhancement, a median filter, and a bilateral filter. We evaluate our method on the KITTI 2012, KITTI 2015, and Middlebury stereo data sets and show that it outperforms other approaches on all three data sets.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, background): “In contrast, recent learning-based solutions to motion estimation [10, 15, 24], stereo matching [20, 21, 42] and 3D reconstruction [5, 6, 9, 16, 37] have demonstrated impressive results by learning feature representations that are much more robust to local viewpoint and lighting changes.”
Learning a Multi-View Stereo Machine (2017, background): “Most recently, CNN based learning methods have been proposed for 3D reconstruction by learning image patch similarity functions [60, 18, 23] and end-to-end disparity regression from stereo pairs [37, 28].”
Unsupervised Learning of Depth and Ego-Motion from Video (2017, background): “, feature matching [18], pose estimation [26], and stereo [10, 27, 53].”
Pyramid Stereo Matching Network (2018, methodology): “Early approaches using CNNs treated the problem of correspondence estimation as similarity computation [27, 30], where CNNs compute the similarity score for a pair of image patches to further determine whether they are matched.”
PatchBatch: A Batch Augmented Loss for Optical Flow (2015, methodology): “Architectures similar to the one described above were explored by previous work [41, 43].”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, methodology): “Zbontar and LeCun [39] and later Zbontar et al. [42] use an encoder composed of four convolutional layers, see Fig.”
Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems (2017, background): “optical flow [16] and stereo matching (disparity estimation) [46, 6, 27].”
Conditional Affordance Learning for Driving in Urban Environments (2018, background): “, for object detection [8, 9], image segmentation [10, 11], depth [12, 13] or motion estimation [14, 15].”
LAF-Net: Locally Adaptive Fusion Networks for Stereo Confidence Estimation (2019, methodology): “For computing the MC-CNN, ‘KITTI 2012 fast network’ was used, provided at the author’s website [46].”
Angel-Eye: A Complete Design Flow for Mapping CNN Onto Embedded FPGA (2018, methodology): “Thus, CNNs can help achieve the top performance in regions like image classiﬁcation [1], [2], object detection [3], and even stereo vision [4].”
DeepMVS: Learning Multi-view Stereopsis (2018, methodology): “For the use of ConvNets in visual reconstruction problems, early work focuses on learning patch similarity for stereo matching [42, 41, 26].”
DDFlow: Learning Optical Flow with Unlabeled Data Distillation (2019, result): “Going forward, our results suggest that our data distillation technique may be a promising direction for advancing other vision tasks like stereo matching (Zbontar and LeCun 2016) or depth estimation (Eigen, Puhrsch, and Fer-gus 2014).”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, neutral): “Our approach is inspired by cost volumes [23, 54, 85], which have proven successful for optical flow.”
Non-Local Spatial Propagation Network for Depth Completion (2020, methodology): “on algorithms often utilize information from a single modality. Eigen et al. [11] used a multi-scale neural network to predict depth from a single image. In the method introduced by Zbontar and LeCun [35], the deep features of image patches are extracted from stereo rectied images, and then the disp”
On Translation Invariance in CNNs: Convolutional Layers Can Exploit Absolute Spatial Location (2020, methodology): “Examples of region cropping in CNNs include: The bounding box in object detection [27, 34, 79]; processing a huge resolution image in patches [42, 86]; local image region matching [32, 66, 108, 107]; local CNN patch pooling encoders [3, 6, 8].”
Real-Time Self-Adaptive Deep Stereo (2018, methodology): “CNN based matching cost functions [44, 5, 20] achieved state-of-the-art on both KITTI and Middlebury v3 by replacing conventional cost functions [14] within the SGM pipeline [13].”
Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference (2019, methodology): “One concern about the classification formulation is the discretized depth map output [32, 20, 13].”