Hierarchical Surface Prediction for 3D Object Reconstruction

By Christian Häne, Shubham Tulsiani, Jitendra Malik. 2017

347 citations recorded by the source corpus.

Recently, Convolutional Neural Networks have shown promising results for 3D geometry prediction. They can make predictions from very little input data such as a single color image. A major limitation of such approaches is that they only predict a coarse resolution voxel grid, which does not capture the surface of the objects well. We propose a general framework, called hierarchical surface prediction (HSP), which facilitates prediction of high resolution voxel grids. The main insight is that it is sufficient to predict high resolution voxels around the predicted surfaces. The exterior and interior of the objects can be represented with coarse resolution voxels. Our approach is not dependent on a specific input type. We show results for geometry prediction from color images, depth images and shape completion from partial voxel grids. Our analysis shows that our high resolution predictions are more accurate than low resolution predictions.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “Due to the high memory requirements of voxel representations, recent works have proposed to reconstruct 3D objects in a multi-resolution fashion [28, 67].”
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation (2019, methodology): “Octree-based methods [49, 41, 22] alleviate the compute and memory limitations of dense voxel methods, extending for example the ability to learn at up to 512 resolution [49], but even this resolution is far from producing shapes that are visually compelling.”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…networks have contributed to the significant progress in recent years, mostly with voxel representations [Choy et al., 2016, Girdhar et al., 2016, Häne et al., 2017, Kar et al., 2015, Novotny et al., 2017, Rezende et al., 2016, Tatarchenko et al., 2016, Tulsiani et al., 2017, Wu et al., 2016,…”
Learning Implicit Fields for Generative Shape Modeling (2018, methodology): “We compare our approach with two state-of-the-art SVR methods, HSP [18], an octree-based method that uses 3D CNN decoders to generate 256(3) voxels, and AtlasNet [16], which warps surface patches onto target shapes.”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “D models. With the development of largescale shape repositories like ShapeNet [5] and methods like deep convolutional networks, researchers have built more scalable and ecient models in recent years [7,13,18,27,36,38,48,52,56,57,62]. While most of these approaches encode 4 J. Wu et al. objects in vo”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, methodology): “, octrees) have been proposed to increase the output resolution beyond 323 voxels [12, 27, 31].”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Using deep convolutional networks, researchers have made significant progress in recent years [9, 17, 21, 29, 41, 43, 56, 59, 60, 62, 66, 52, 61].”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “Relying on ground-truth 3D supervision (using synthetic data), some approaches have examined learning voxel [4,8,39,33], point cloud [7] or octree [10,26] prediction.”
A Papier-Mache Approach to Learning 3D Surface Generation (2018, neutral): “To overcome this, recent work has looked at a voxel representation of the surface of a shape via oct-trees [12, 26, 30].”
Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, methodology): “We compare against HSP [9] using the first dataset DataHSP , and against OGN [43] using the second dataset Data3D−R2N2.”
Learning a Multi-View Stereo Machine (2017, background): “We can borrow ideas from very recent works [43, 21] which show that CNNs can predict such high resolution volumes.”
The limits and potentials of deep learning for robotics (2018, background): “Similarly, there is a lot of ongoing work on estimating the 3D structure of objects from a single or multiple views without having depth information available [20], [40], [131], [137].”
Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction (2020, methodology): “Octree-based methods [50,43,24] relax the compute and memory limitations of dense voxel methods to some degree and have been shown on voxel resolutions of up to 512.”
Learning 3D Shape Completion from Laser Scan Data with Weak Supervision (2018, background): “Some approaches [24, 39, 45] use octrees to predict high-resolution shapes via supervision provided at multiple scales.”
Convolutional Occupancy Networks (2020, background): “ed 3D reconstruction [5,46,47]. Due to the cubic memory requirements of voxel-based representations, several works proposed to operate on multiple scales or use octrees for ecient space partitioning [9,16,26,38,39,43]. However, even when using adaptive data structures, voxel-based techniques are sti”
HyperCube: Implicit Field Representations of Voxelized 3D Models (2021, background): “3D objects can be represented using different approaches including voxel grids [4, 5, 6, 7], octrees [8, 9, 10, 11], multi-view images [12, 13, 14], point clouds [15, 16, 17, 18, 19], geometry images [20, 21], deformable meshes [5, 21, 22, 19], and part-based structural graphs [23, 24].”
What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction (2021, background): “Most work on invisible surfaces focuses on single objects with voxels [9, 18, 21], point-clouds [14, 32], CAD models [26] and meshes [19, 20].”
Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction (2017, background): “Concurrent works follow to use similar concepts [8, 25] to predict 3D volumetric data with higher granularity.”