Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
By Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X. Chang, Matthias Nießner. 2018
284 citations recorded by the source corpus.
We present Scan2CAD, a novel data-driven method that learns to align clean 3D CAD models from a shape database to the noisy and incomplete geometry of a commodity RGB-D scan. For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry. To tackle this problem, we create a new scan-to-CAD alignment dataset based on 1506 ScanNet scans with 97607 annotated keypoint pairs between 14225 CAD models from ShapeNet and their counterpart objects in the scans. Our method selects a set of representative keypoints in a 3D scan for which we find correspondences to the CAD geometry. To this end, we design a novel 3D CNN architecture that learns a joint embedding between real and synthetic objects, and from this predicts a correspondence heatmap. Based on these correspondence heatmaps, we formulate a variational energy minimization that aligns a given set of CAD models to the reconstruction. We evaluate our approach on our newly introduced Scan2CAD benchmark where we outperform both handcrafted feature descriptor as well as state-of-the-art CNN based methods by 21.39%.
Subfields: Aerospace Engineering, Computational Mechanics, Computer Graphics and Computer-Aided Design, Geology, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
- Matthias Nießner
- Leonidas J. Guibas
- Angela Dai
- Charles Ruizhongtai Qi
- Or Litany
- Silvio Savarese
- Jitendra Malik
- Vladlen Koltun
In Computational Mechanics
- Matthias Nießner
- Leonidas J. Guibas
- Angela Dai
- Or Litany
- Charles Ruizhongtai Qi
- Andreas Geiger
- Thomas A. Funkhouser
- Silvio Savarese
In Computer Graphics and Computer-Aided Design
- Matthias Nießner
- Leonidas J. Guibas
- Thomas A. Funkhouser
- Angela Dai
- Andreas Geiger
- Kalyan Sunkavalli
- Angel X. Chang
- Manolis Savva
In Geology
- Matthias Nießner
- Angela Dai
- Or Litany
- Leonidas J. Guibas
- Charles Ruizhongtai Qi
- Silvio Savarese
- Angel X. Chang
- Jiwen Lu
In 3D Vision & Pose
- Matthias Nießner
- Angela Dai
- Leonidas J. Guibas
- Charles Ruizhongtai Qi
- Or Litany
- Andreas Geiger
- Thomas A. Funkhouser
- Jitendra Malik
Citations in the corpus
- Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, methodology): “ReplicaCAD like other efforts such as OpenRooms [61] and Scan2cad [62] creates a version of the scene with decoupled 3D models suited for physical simulation and interaction.”
- OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets (2020, methodology): “Previously, Scan2cad [2] has provided the ground-truth poses that aligned CAD models from ShapeNet [11] to the ScanNet point clouds.”
- Interactive Gibson Benchmark: A Benchmark for Interactive Navigation in Cluttered Environments (2019, methodology): “Available: https: //github.com/pybind/pybind11 [48] A. Avetisyan et al., “Scan2CAD: Learning CAD model alignment in RGBD scans,” in Proc.”
- 3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics (2020, methodology): “from Scan Raw Mesh Scan2CAD [5] Real scan 1,506 3,049 No texture Mesh OpenRooms [24] Real scan 1,068 2,500 Amateur Mesh”
- ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes (2020, methodology): “For (a) we utilized the Scan2CAD [14] annotations that provide 9DOF alignments between ShapeNet models and ScanNet objects, and for (b) we used a combination of PartNet’s [45] and manual annotations.”
- Back to Reality: Weakly-supervised 3D Object Detection with Shape-guided Label Enhancement (2022, methodology): “Existing approaches can be divided into two categories: supervised [2, 3, 6, 40] and unsupervised [8, 19, 24, 30, 42].”
- ImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes (2020, methodology): “To locate objects using purely geometric information, one popular line of methods is based on template matching using a collection of clean CAD models either directly [21, 28, 25], or through extracted features [44, 2].”
- MonoScene: Monocular 3D Semantic Scene Completion (2021, methodology): “Subsequently, many algorithms use dedicated depth sensors such as Lidar [36, 50, 62] or depth cameras [2, 15, 19], easing the 3D estimation problem.”
- OpenRooms: An Open Framework for Photorealistic Indoor Scene Datasets (2020, methodology): “While recent methods such as [6] are possible alternatives, we demonstrate our dataset creation example utilizing existing labels in ScanNet and initial CAD alignment [5] to create the ground truth geometry robustly.”
- Local Implicit Grid Representations for 3D Scenes (2020, methodology): “…with a coarse-to-fine inference strategy to directly regress gridded Truncated Signed Distance Function (TSDF) outputs from incomplete input TSDF. [1] tackles the scene reconstruction problem by CAD model retrieval, which produces attractive surfaces, at the expense of geometric inaccuracies.”
- TEASER: Fast and Certifiable Point Cloud Registration (2020, methodology): “Scan2CAD [91] and its improvement [92] apply similar pipelines to align CAD models to RGB-D scans.”
- Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve (2020, methodology): “etrieving and aligning similar CAD models. Most methods focus on aligning CAD models to RGB-D scan, point cloud geometry, or 2D-3D surface mapping though various geometric feature matching techniques [1,13,19,21,24,37]. Izadinia and Seitz [20] and Huang et al. [17] propose to optimize for both scene”