PatchMatch Stereo - Stereo Matching with Slanted Support Windows

By Michael Bleyer, Christoph Rhemann, Carsten Rother. 2011

810 citations recorded by the source corpus.

Common local stereo methods match support windows at integer-valued disparities. The implicit assumption that pixels within the support region have constant disparity does not hold for slanted surfaces and leads to a bias towards reconstructing fronto-parallel surfaces. This work overcomes this bias by estimating an individual 3D plane at each pixel onto which the support region is projected. The major challenge of this approach is to ﬁnd a pixel’s optimal 3D plane among all possible planes whose number is inﬁnite. We show that an ideal algorithm to solve this problem is PatchMatch [1] that we extend to ﬁnd an approximate nearest neighbor according to a plane. In addition to Patch-Match’s spatial propagation scheme, we propose (1) view propagation where planes are propagated among left and right views of the stereo pair and (2) temporal propagation where planes are propagated from preceding and consecutive frames of a video when doing temporal stereo. Adaptive support weights are used in matching cost aggregation to improve results at disparity borders. We also show that our slanted support windows can be used to compute a cost volume for global stereo methods, which allows for explicit treatment of occlusions and can handle large untextured regions. In the results we demonstrate that our method reconstructs highly slanted surfaces and achieves impressive disparity details with sub-pixel precision. In the Middlebury table, our method is currently top-performer among local methods and takes rank 2 among approximately 110 competitors if sub-pixel precision is considered.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

IRS: A Large Synthetic Indoor Robotics Stereo Dataset for Disparity and Surface Normal Estimation (2019, methodology): “Compared to monocular disparity estimation and normal estimation which rely on the prior information of the scene and lack geometric constraints, stereo vision can combine the prior information and geometry information together to give better estimated results [7], [8], [9].”
SurfaceNet: An End-to-End 3D Neural Network for Multiview Stereopsis (2017, methodology): “The Gipuma algorithm proposed in [9] is a massively parallel method for multiview matching built on the idea of patchmatch stereo [3].”
Kornia: an Open Source Differentiable Computer Vision Library for PyTorch (2019, background): “Multi-view reconstruction is a well understood problem with a good geometric model [21], and many approaches for matching and optimization [39, 8, 51], and some recent promising deep learning approaches [34].”
A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos (2017, methodology): “For this scenario, we evaluate the popular patch-based PMVS [4], Gipuma [5], which is a well-performing PatchMatch-based variant [26], the multi-view stereo method based on pixelwise view selection in COLMAP [35], and CMPMVS [16], which aims at reconstructing weakly supported surfaces.”
HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching (2020, methodology): “Efficient algorithms [31, 34, 5, 3] avoid searching the full disparity space by using patchmatch [1] and super-pixel [31] techniques.”
UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction (2021, background): “Before the era of deep learning, classic multi-view stereo (MVS) methods [2, 4, 5, 7, 21, 21, 48, 50, 51] focus on either matching features across views [4, 48] or representing shapes with a voxel grid [2, 5, 7, 21, 28, 42, 50, 54, 55].”
Fast cost-volume filtering for visual correspondence and beyond (2011, background): “[49] proposed to overcome this problem by estimating a 3D plane at each pixel onto which the support region is projected.”
SphereFlow: 6 DoF Scene Flow from RGB-D Pairs (2014, background): “[5] treat views sequentially, with the effect of carrying out their form of view propagation after a full iteration is completed.”
Multi-View Stereo: A Tutorial (2015, background): “Notably, “PatchMatch Stereo” is a successful algorithm [40], which randomly initializes depth values, then refines them based on the local propagation and the random search strategies.”
DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch (2019, methodology): “solution space and the use of sophisticated post-processing procedures. A plethora of acceleration approaches have been proposed toward achieving real-time performance, such as solution space pruning [6], coarse-to-ﬁne cost volume computation [16] and efﬁcient variational optimization [10].Among the”
Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding (2018, methodology): “Geometric based methods estimate 3D from a given video with feature matching or patch matching, such as PatchMatch Stereo [26], SfM [13], SLAM [27], [28] and DTAM [1], and are effective and efficient inmany cases.”
High-Accuracy Stereo Matching Based on Adaptive Ground Control Points (2015, methodology): “In this paper, the local plane model [22] is used, which is defined as F p (q) = A p (q x − p x ) + B p (q y − p y ), (6) where A p and B p are the plane parameters.”
Just Look at the Image: Viewpoint-Specific Surface Normal Prediction for Improved Multi-View Reconstruction (2016, methodology): “Notable examples include the well-known PMVS method [16], as well as the multi-view variant [18] of the PatchMatch stereo algorithm [5].”
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019, methodology): “Classic multi-view stereo (MVS) methods [5–7,20,37,60,68–70] usually match features between neighboring views [5, 20, 68] or reconstruct the 3D shape in a voxel grid [6, 7, 37, 60, 70].”
PatchmatchNet: Learned Multi-View Patchmatch Stereo (2020, methodology): “Unlike [3, 16, 38], we refrain from parameterizing the per-pixel hypothesis as a slanted plane, due to heavy memory penalties.”