3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions

By Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, Thomas A. Funkhouser. 2016

1,219 citations recorded by the source corpus.

Matching local geometric features on real-world depth images is a challenging task due to the noisy, low-resolution, and incomplete nature of 3D scan data. These difficulties limit the performance of current state-of-art methods, which are typically based on histograms over geometric properties. In this paper, we present 3DMatch, a data-driven model that learns a local volumetric patch descriptor for establishing correspondences between partial 3D data. To amass training data for our model, we propose a self-supervised feature learning method that leverages the millions of correspondence labels found in existing RGB-D reconstructions. Experiments show that our descriptor is not only able to match local geometry in new scenes for reconstruction, but also generalize to different tasks and spatial scales (e.g. instance-level object model alignment for the Amazon Picking Challenge, and mesh surface correspondence). Results show that 3DMatch consistently outperforms other state-of-the-art approaches by a significant margin. Code, data, benchmarks, and pre-trained models are available online at http://3dmatch.cs.princeton.edu.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation (2019, background): “Voxel-based SDF representations have been extensively used for 3D shape learning [59, 16, 49], but their use of discrete voxels is expensive in memory.”
Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning (2018, methodology): “A common approach to deploying these methods in real systems involves pre-computing grasps from a database of known 3D object models [23], and indexing them at run-time with point cloud registration for object pose estimation [24], [25].”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, methodology): “The SUN3D ground truth correspondences and reg-istrations are obtained from [15], using the training and testing scenes split from [46].”
TossingBot: Learning to Throw Arbitrary Objects With Residual Physics (2019, background): “…3 other aspects that enable our system’s picking speeds: 1) fast algorithmic run-time speeds (220ms for inference), 2) real-time TSDF fusion [5, 21, 26] of RGB-D data, which enables us to capture and aggregate observed 3D data of the scene simultaneously as the robot moves around within the…”
Learning Visual Affordances for Robotic Manipulation (2019, methodology): “The second uses more elaborated local descriptors such as SIFT keypoints [23] for color data or 3DMatch [122] for 3D data.”
Image Matching from Handcrafted to Deep Features: A Survey (2020, methodology): “Zeng et al. (2017) proposed to use a 3D deep CNN for learning a local volumetric patch descriptor.”
DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision (2023, unknown): “It also aids in separating viewpoint effects in learning view-invariant representations like patch descriptors [40] and normals [41].”
Deep Closest Point: Learning Representations for Point Cloud Registration (2019, methodology): “Recently, descriptor learning methods have brought significant progress in point cloud registration: 3DMatch [59] proposes learning a local volumetric patch descriptor to establish correspondences; 3DFeatNet [56] takes similar approach to point cloud representation for local regions; PPFFoldNet [9] ”
PREDATOR: Registration of 3D Point Clouds with Low Overlap (2020, methodology): “Following the literature [49, 17, 8], we also report Feature Match Recall (FMR), defined as the fraction of pairs that have >5% ”inlier” matches with <10 cm residual under the ground truth transformation (without checking if the transformation can be recovered from those matches), and Inlier Ratio (”
A comprehensive survey on point cloud registration (2021, methodology): “For example, [114] uses AlexNet to learn a 3D feature from an RGB-D dataset.”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, methodology): “Learning DS-NeRF with the RGB-D reconstruction output [34] further improves performance, highlighting the potential of applying our method alongside other sources of depth.”
Rotation Invariant Convolutions for 3D Point Clouds Deep Learning (2019, background): “In addition, there are a few previous works about learning local descriptors from point clouds for feature matching [43, 14, 7], some of which [7] can be rotation invariant.”
D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features (2020, methodology): “For metric 2), we use a RANSAC with 50,000 max iterations (following [36]) to estimate the transformation matrics.”
Geometric Transformer for Fast and Robust Point Cloud Registration (2022, methodology): “Extensive experiments on both indoor and outdoor benchmarks [13,37] demonstrate the efficacy of GeoTransformer.”