Scan2CAD: Learning CAD Model Alignment in RGB-D Scans

By Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X. Chang, Matthias Nießner. 2018

284 citations recorded by the source corpus.

We present Scan2CAD, a novel data-driven method that learns to align clean 3D CAD models from a shape database to the noisy and incomplete geometry of a commodity RGB-D scan. For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry. To tackle this problem, we create a new scan-to-CAD alignment dataset based on 1506 ScanNet scans with 97607 annotated keypoint pairs between 14225 CAD models from ShapeNet and their counterpart objects in the scans. Our method selects a set of representative keypoints in a 3D scan for which we find correspondences to the CAD geometry. To this end, we design a novel 3D CNN architecture that learns a joint embedding between real and synthetic objects, and from this predicts a correspondence heatmap. Based on these correspondence heatmaps, we formulate a variational energy minimization that aligns a given set of CAD models to the reconstruction. We evaluate our approach on our newly introduced Scan2CAD benchmark where we outperform both handcrafted feature descriptor as well as state-of-the-art CNN based methods by 21.39%.

Subfields: Aerospace Engineering, Computational Mechanics, Computer Graphics and Computer-Aided Design, Geology, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

  1. Matthias Nießner
  2. Leonidas J. Guibas
  3. Angela Dai
  4. Charles Ruizhongtai Qi
  5. Or Litany
  6. Silvio Savarese
  7. Jitendra Malik
  8. Vladlen Koltun

In Computational Mechanics

  1. Matthias Nießner
  2. Leonidas J. Guibas
  3. Angela Dai
  4. Or Litany
  5. Charles Ruizhongtai Qi
  6. Andreas Geiger
  7. Thomas A. Funkhouser
  8. Silvio Savarese

In Computer Graphics and Computer-Aided Design

  1. Matthias Nießner
  2. Leonidas J. Guibas
  3. Thomas A. Funkhouser
  4. Angela Dai
  5. Andreas Geiger
  6. Kalyan Sunkavalli
  7. Angel X. Chang
  8. Manolis Savva

In Geology

  1. Matthias Nießner
  2. Angela Dai
  3. Or Litany
  4. Leonidas J. Guibas
  5. Charles Ruizhongtai Qi
  6. Silvio Savarese
  7. Angel X. Chang
  8. Jiwen Lu

In 3D Vision & Pose

  1. Matthias Nießner
  2. Angela Dai
  3. Leonidas J. Guibas
  4. Charles Ruizhongtai Qi
  5. Or Litany
  6. Andreas Geiger
  7. Thomas A. Funkhouser
  8. Jitendra Malik

Citations in the corpus