SurfNet: Generating 3D Shape Surfaces Using Deep Residual Networks

By Ayan Sinha, Asim Unmesh, Qi-Xing Huang, Karthik Ramani. 2017

187 citations recorded by the source corpus.

3D shape models are naturally parameterized using vertices and faces, i.e., composed of polygons forming a surface. However, current 3D learning paradigms for predictive and generative tasks using convolutional neural networks focus on a voxelized representation of the object. Lifting convolution operators from the traditional 2D to 3D results in high computational overhead with little additional benefit as most of the geometry information is contained on the surface boundary. Here we study the problem of directly generating the 3D shape surface of rigid and non-rigid shapes using deep convolutional neural networks. We develop a procedure to create consistent ‘geometry images’ representing the shape surface of a category of 3D objects. We then use this consistent representation for category-specific shape surface generation from a parametric representation or an image by developing novel extensions of deep residual networks for the task of geometry image generation. Our experiments indicate that our network learns a meaningful representation of shape surfaces allowing it to interpolate between shape orientations and poses, invent new shape surfaces and reconstruct 3D shape surfaces from previously unseen images. Our code is available at https://github.com/sinhayan/surfnet.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Generative Models

In 3D Vision & Pose

Citations in the corpus

Learning Implicit Fields for Generative Shape Modeling (2018, background): “There have been a variety of 3D shape representations for deep learning of shapes, such as voxel grids [9, 14, 28, 45, 46], octrees [18, 35, 40, 43, 44], multi-view images [29, 39], point clouds [1, 12, 13, 32, 33, 47, 48], geometry images [37, 38], deformable mesh/patches [16, 38, 42, 47], and part”
Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images (2018, methodology): “Besides these typical 3D representations, there is an interesting work [28] which uses a so-called “geometry image” to represent a 3D shape.”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, background): “While some learning based methods do pursue mesh prediction [14,35,18,24], they also rely on 3D supervision which is only available for restricted classes or in a synthetic setting.”
A Papier-Mache Approach to Learning 3D Surface Generation (2018, neutral): “Recently, Sinha et al. [29] proposed to use a spherical parameterization of a single deformable mesh (if available) or of a few base shapes (composed with authalic projection of a sphere to a plane) to represent training shapes as parameterized meshes.”
Deep Learning 3D Shapes Using Alt-az Anisotropic 2-Sphere Convolution (2018, methodology): “A straightforward extension is to treat a 3D shape as a voxel grid (Wu et al. (2015); Maturana & Scherer (2015); Song & Xiao (2016); Wang et al.”
TopNet: Structural Point Cloud Decoder (2019, background): “Although moderate success has been achieved with performing shape completion on regular representations of 3D objects such as distance fields, meshes, and voxel grids [3, 6, 22, 28, 16, 14], these methods fall short due to inefficiency of such representations.”
Multi-chart generative surface modeling (2018, methodology): “We note that [29] solve for dense correspondences of the input shapes as preprocess, which is a challenging problem that currently cannot always be accomplished with high accuracy.”
Articulation-Aware Canonical Surface Mapping (2020, background): “Several previous approaches similarly pursue pixel to surface [2, 23, 24, 28, 42] or volume [35] mappings, but unlike our approach, crucially rely on direct supervision towards this end.”
HyperCube: Implicit Field Representations of Voxelized 3D Models (2021, background): “3D objects can be represented using different approaches including voxel grids [4, 5, 6, 7], octrees [8, 9, 10, 11], multi-view images [12, 13, 14], point clouds [15, 16, 17, 18, 19], geometry images [20, 21], deformable meshes [5, 21, 22, 19], and part-based structural graphs [23, 24].”
Im2Vec: Synthesizing Vector Graphics without Vector Supervision (2021, background): “Sinha et al. [34] represents watertight 3D shapes as continuous deformation of a sphere.”
SAL: Sign Agnostic Learning of Shapes From Raw Data (2019, methodology): “[32, 33] use geometry images as global parameterizations, while [27] use conformal global parameterizations to reduce the number of degrees of freedom of the map.”
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, background): “Recent work also tried to directly reconstruct surface meshes [34, 25, 53, 19, 46, 28], deformable shapes [24, 25], and some learned implicit distance functions [7, 41, 37, 6].”
Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation (2019, background): “Some other representations have been proposed, such as geometry images [35], depth images [40, 33], classification boundaries [28, 4], signed distance function [30], etc.”
Deep Volumetric Video From Very Sparse Multi-view Performance Capture (2018, background): “More closely related, a number of previous works apply multi-view CNNs to 3D reconstruction problems in both unsupervised [65] and supervised approaches to obtain the ﬁnal geometry directly [66, 67], or indirectly via normal maps [68], silhouettes [69], or color images [70].”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, methodology): “SurfNet [45] learns a 3D offset from a template UV map of a surface.”
DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction (2019, background): “[23] propose to generate the surface of an object using geometry images.”
Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network (2018, unknown): “5D geometry(height map)[41, 42], 3D geometry(geometry image)[21, 54] and the correspondences between 3D facial meshes[7].”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, background): “Sinha et al. [52] proposed a cut-invariant procedure, which solves a large-scale correspondence problem, and an extension of deep residual nets to automatically generate geometry images encoding the x;y;z surface coordinates.”
Implicit Geometric Regularization for Learning Shapes (2020, background): “Some works have considered global surface parametrizations (Sinha et al., 2016; 2017; Maron et al., 2017).”
BSP-Net: Generating Compact Meshes via Binary Space Partitioning (2019, methodology): “Amongst these we find a category of techniques pioneered by PointNet [33] that express surfaces as point clouds [1, 9, 10, 33, 34, 53, 56], and techniques pioneered by AtlasNet [16] that adopt a 2D-to-3D mapping process [47, 39, 44, 53].”