Learned Multi-patch Similarity

By Wilfried Hartmann, Silvano Galliani, Michal Havlena, Luc Van Gool, Konrad Schindler. 2017

113 citations recorded by the source corpus.

Estimating a depth map from multiple views of a scene is a fundamental task in computer vision. As soon as more than two viewpoints are available, one faces the very basic question how to measure similarity across >2 image patches. Surprisingly, no direct solution exists, instead it is common to fall back to more or less robust averaging of two-view similarities. Encouraged by the success of machine learning, and in particular convolutional neural networks, we propose to learn a matching function which directly maps multiple image patches to a scalar similarity score. Experiments on several multi-view datasets demonstrate that this approach has advantages over methods based on pairwise patch similarity.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, methodology): “CNNs have been proven successful for learning similarity measures between two or more image patches for stereo [20, 40, 41] as well as multi-view stereo [14].”
Learning a Multi-View Stereo Machine (2017, background): “Most recently, CNN based learning methods have been proposed for 3D reconstruction by learning image patch similarity functions [60, 18, 23] and end-to-end disparity regression from stereo pairs [37, 28].”
Deep Volumetric Video From Very Sparse Multi-view Performance Capture (2018, background): “Hartmann et al. [73] propose a deep learning based approach to predict the similarity between the image patches across multiple views, which enables 3D reconstruction using stereopsis.”
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, background): “A depth map represents dense 3D information that is perfectly aligned with a reference view; depth reconstruction has been demonstrated in many previous works on reconstruction with both single view [9, 51, 16, 17, 62] and multiple views [4, 48, 18, 14, 43, 57, 43].”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, methodology): “Learning-based MVS methods usually replace some parts of the classic MVS pipeline, e.g., feature matching [23, 36, 40, 74, 88], depth fusion [16,57], or inferring depth from multi-view images [29,79,80,85].”
UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction (2021, background): “For instance, some works learn to match 2D features [15, 22, 27, 56, 63], fuse depth maps [11, 46], or infer depth maps [16, 58, 59] from multi-view images.”
Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (2019, background): “Multi-patch similarity [9] introduces a learned cost metric.”
DeepMVS: Learning Multi-view Stereopsis (2018, methodology): “A line of work focuses on learning a good similarity measure for patch matching across two views [42] and multiple views [15] using ConvNets.”
MVSNet: Depth Inference for Unstructured Multi-view Stereo (2018, methodology): “propose the learned multi-patch similarity [11] to replace the traditional cost metric for MVS reconstruction.”
BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks (2019, background): “Learned multi-patch similarity [7] ﬁrst applies deep neural networks for MVS cost metrics learning.”
Deep learning in optical metrology: a review (2022, background): “Hartmann et al.(338) constructed a CNN with five siamese branches to learn a matching function, which could directly predict a scalar similarity score from multiple image patches.”
Cost Volume Pyramid Based Depth Inference for Multi-View Stereo (2019, background): “More recently, a few approaches have proposed to directly solve MVS problems [14, 24, 29].”
Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference (2019, background): “Multi-patch similarity [11] is proposed to replace the traditional cost metric with the learned one.”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, methodology): “Hartmann et al. [47], on the other hand, aggregate the features coming from multiple patches using mean pooling before feeding them to a decision network.”