3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction
By Christopher Bongsoo Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, Silvio Savarese. 2016
1,900 citations recorded by the source corpus.
Inspired by the recent success of methods that employ shape priors to achieve robust 3D reconstructions, we propose a novel recurrent neural network architecture that we call the 3D Recurrent Reconstruction Neural Network (3D-R2N2). The network learns a mapping from images of objects to their underlying 3D shapes from a large collection of synthetic data [13]. Our network takes in one or more images of an object instance from arbitrary viewpoints and outputs a reconstruction of the object in the form of a 3D occupancy grid. Unlike most of the previous works, our network does not require any image annotations or object class labels for training or testing. Our extensive experimental analysis shows that our reconstruction framework (i) outperforms the state-of-the-art methods for single view reconstruction, and (ii) enables the 3D reconstruction of objects in situations when traditional SFM/SLAM methods fail (because of lack of texture and/or wide baseline).
Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
- Jitendra Malik
- Andreas Geiger
- Silvio Savarese
- Leonidas J. Guibas
- Matthias Nießner
- Vladlen Koltun
- Kristen Grauman
- Abhinav Kumar Gupta
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Andrea Vedaldi
- Xiaogang Wang
- Andrew Zisserman
In Geology
- Matthias Nießner
- Leonidas J. Guibas
- Jitendra Malik
- Angela Dai
- Andreas Geiger
- Or Litany
- Charles Ruizhongtai Qi
- Silvio Savarese
In 3D Vision & Pose
- Andreas Geiger
- Jitendra Malik
- Michael Niemeyer
- Jiajun Wu
- Matthias Nießner
- Justin Johnson
- Gerard Pons-Moll
- Josh Tenenbaum
Citations in the corpus
- Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, methodology): “[9].”
- A Point Set Generation Network for 3D Object Reconstruction from a Single Image (2016, methodology): “As stated in [5], their method often misses thin features of objects (e.”
- RealPoint3D: Point Cloud Generation from a Single Image with Complex Background (2018, methodology): “All rights reserved. achieved great success for object classification, detection and segmentation (Cicek et al. 2016; Chen et al. 2016; Meagher 1980; Wu et al. 2016; Choy et al. 2016; Mat-urana and Scherer 2015; Tatarchenko, Dosovitskiy, and Brox 2017).”
- OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “Most existing 3D network architectures [8,29,34,47] replace the 2D pixel array by its 3D analogue, i.”
- Deep Level Sets: Implicit Surface Representations for 3D Shape Inference (2019, methodology): “[3] introduced generative adversarial models (GANs) using long short-term memory (LSTM) for reconstructing voxels or point clouds achieving state-of-the-art results.”
- Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…repositories [Chang et al., 2015] and deep networks have contributed to the significant progress in recent years, mostly with voxel representations [Choy et al., 2016, Girdhar et al., 2016, Häne et al., 2017, Kar et al., 2015, Novotny et al., 2017, Rezende et al., 2016, Tatarchenko et al., 2016,…”
- Learning Implicit Fields for Generative Shape Modeling (2018, background): “There have been a variety of 3D shape representations for deep learning of shapes, such as voxel grids [9, 14, 28, 45, 46], octrees [18, 35, 40, 43, 44], multi-view images [29, 39], point clouds [1, 12, 13, 32, 33, 47, 48], geometry images [37, 38], deformable mesh/patches [16, 38, 42, 47], and part”
- Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “researchers have made impressive progress on the these tasks [7,52,8], making use of gigantic 3D datasets [5,60,59].”
- Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images (2018, methodology): “We use the dataset provided by Choy et al. [6].”
- Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “Relying on ground-truth 3D supervision (using synthetic data), some approaches have examined learning voxel [4,8,39,33], point cloud [7] or octree [10,26] prediction.”
- A Papier-Mache Approach to Learning 3D Surface Generation (2018, positive): “We used the rendered views provided by [6] and sampled 3D points on the shapes using [32].”
- Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, methodology): “Implementation For 3D object reconstruction, we first trained a standard autoencoder, similar to prior work [3, 41], to produce objects at 32(3) resolution.”
- MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, methodology): “For quantitative evaluation, previous works usually compute the Intersection-over-Union (IoU) [Tulsiani et al., 2017, Choy et al., 2016].”
- SurfaceNet: An End-to-End 3D Neural Network for Multiview Stereopsis (2017, background): “While the 3D reconstruction of an object from a single or multiple views using convolutional neural networks has been studied in [26, 5], the approaches focus on a very coarse reconstruction without geometric details when only one or very few images are available.”
- Learning a Multi-View Stereo Machine (2017, methodology): “More specifically, we use a subset of 13 major categories (same as [5]) containing around 44k 3D models resized to lie within the unit cube centered at the origin with a train/val/test split of [0.”
- OctNetFusion: Learning Depth Fusion from Data (2017, background): “By utilizing large datasets of 3D models and high capacity 3D convolutional neural networks (3D CNNs), our approach learns to smooth out sensor noise, deals with outliers and completes missing 3D geometry.”