Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve
By Weicheng Kuo, Anelia Angelova, Tsung-Yi Lin, Angela Dai. 2020
88 citations recorded by the source corpus.
Object recognition has seen significant progress in the image domain, with focus primarily on 2D perception. We propose to leverage existing large-scale datasets of 3D models to understand the underlying 3D structure of objects seen in an image by constructing a CAD-based representation of the objects and their poses. We present Mask2CAD, which jointly detects objects in real-world images and for each detected object, optimizes for the most similar CAD model and its pose. We construct a joint embedding space between the detected regions of an image corresponding to an object and 3D CAD models, enabling retrieval of CAD models for an input RGB image. This produces a clean, lightweight representation of the objects in an image; this CAD-based representation ensures a valid, efficient shape representation for applications such as content creation or interactive scenarios, and makes a step towards understanding the transformation of real-world imagery to a synthetic domain. Experiments on real-world images from Pix3D demonstrate the advantage of our approach in comparison to state of the art. To facilitate future research, we additionally propose a new image-to-3D baseline on ScanNet which features larger shape diversity, real-world occlusions, and challenging image views.
Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Jitendra Malik
- Angjoo Kanazawa
- Ilija Radosavovic
- Zhe Cao
- Donald Metzler
- Simon Stent
- Mostafa Dehghani
- Jianbo Shi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Cordelia Schmid
- Andrew Zisserman
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Andrea Vedaldi
In Geology
In 3D Vision & Pose
- Angjoo Kanazawa
- Jitendra Malik
- Ilija Radosavovic
- Simon Stent
- Zhe Cao
- Jianbo Shi
- Lingzhi Zhang
- Shenghao Zhou
Citations in the corpus
- Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
- Retrieval-Enhanced Machine Learning (2022, methodology): “Related approaches have been also used in computer vision [20, 35, 40, 54].”
- Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “Some approaches [50, 21, 11, 22] utilize neural network to predict the object shape, translation, and global orientation in the camera coordinate.”