Computing the stereo matching cost with a convolutional neural network

By Jure Zbontar, Yann LeCun. 2014

824 citations recorded by the source corpus.

We present a method for extracting depth information from a rectified image pair. We train a convolutional neural network to predict how well two image patches match and use it to compute the stereo matching cost. The cost is refined by cross-based cost aggregation and semiglobal matching, followed by a left-right consistency check to eliminate errors in the occluded regions. Our stereo method achieves an error rate of 2.61% on the KITTI stereo dataset and is currently (August 2014) the top performing method on this dataset.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

SurfaceNet: An End-to-End 3D Neural Network for Multiview Stereopsis (2017, methodology): “For instance, [32] uses a CNN instead of hand-crafted features for finding correspondences among image pairs and [10] predicts normals for depth maps using a CNN, which improves the depth map fusion.”
FlowNet: Learning Optical Flow with Convolutional Networks (2015, background): “Zbontar and LeCun [36] train a CNN with a Siamese architecture to predict similarity of image patches.”
Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2017, background): “Such within-view neighborhood preservation constraints have been extensively explored in the metric learning literature [38], [39], [40], [41], [42], [43].”
Deep Stereo: Learning to Predict New Views from the World's Imagery (2015, neutral): “Unlike this prior work, we learn to synthesize new views directly using a new deep architecture, and do not require known depth or disparity as training data.”
Learning to compare image patches via convolutional neural networks (2015, methodology): “It is worth noting that all costs above can be computed a lot more efficiently using speed optimizations similar with [28].”
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue (2016, methodology): “Methods which do stereo with CNN: Recently, many methods have used CNN to learn good visual features for matching patches which are sampled from the stereo dataset like KITTI [23,3] and match these features while doing classical stereo to get state of the art depth estimation.”
Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches (2015, methodology): “Third place on the leaderboard is held by our previous work (ˇZbontar and LeCun, 2015) with an error rate of 2.61 %.”
Generic 3D Representation via Pose Estimation and Matching (2016, background): “Therefore, we presume (and empirically observe) that the generality of our 3D representation is mostly attributed to the camera pose estimation component. scribing, and matching image features, either through a handcrafting the feature [35,11,39,38,5,40] or learning it [50,25,65,9,66,49,19].”
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions (2016, background): “For instance, various works [33, 32, 42, 17, 43, 17] learn non-linear mappings from local image patches to feature descriptors.”
Group-Wise Correlation Stereo Network (2019, methodology): “Besides hand-crafted methods, researchers also proposed many learned matching costs [29, 13, 21] and cost aggregation algorithms [1, 19].”
SDC – Stacked Dilated Convolution: A Unified Descriptor Network for Dense Matching Tasks (2019, methodology): “This is the exact list of sequences for each subset: • Training: 0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 19, 20, 21, 22, 24, 25, 27, 28, 29, 30, 31, 33, 34, 35, 37, 38, 39, 40, 41, 43, 44, 45, 47 178, 179, 180, 182, 183, 185, 187, 188, 189, 191, 194, 195, 197, 198, 199”
Attention Concatenation Volume for Accurate and Efficient Stereo Matching (2022, background): “Recently, CNN-based stereo models [7, 9, 12, 14, 21, 23] have achieved impressive performance on most of the standard benchmarks.”
The Fast Bilateral Solver (2015, methodology): “Depth maps (1a) produced by the state-of-the-art CNN-based stereo method [36] are processed by our robust bilateral solver using a reference RGB image (1c) and an estimated confidence of that depth map (1b).”
MatchNet: Unifying feature and metric learning for patch-based matching (2015, methodology): “A recent preprint on learning a network for stereo matching has also used the two-tower-plus-fully-connected comparisonnetwork approach [37].”
3DMatch: Learning the Matching of Local 3D Geometry in Range Scans (2016, methodology): “More recently, it has been proposed to use a deep convolutional neutral network to jointly learn the descriptor and the comparison metric for local 2D RGB patches [38, 13].”
Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (2019, methodology): “Learning-based multi-view stereo [4, 44, 45] and stereo matching [3,8,15,46,48] construct 3D cost volumes to measure the similarity between corresponding image patches and determine whether they are matched.”
Learning by Tracking: Siamese CNN for Robust Target Association (2016, background): “In relation to our problem, they have also been remarkably successful in assessing the similarity of image patches for different tasks such as optical flow estimation [21], face verification [49], and depth estimation from multiple viewpoints [22, 60, 62].”
LAF-Net: Locally Adaptive Fusion Networks for Stereo Confidence Estimation (2019, methodology): “For decades, numerous methods have been proposed for this task by leveraging handcrafted [43, 10] and/or machine learning based [45, 38] techniques.”