Neural 3D Mesh Renderer

By Hiroharu Kato, Yoshitaka Ushiku, Tatsuya Harada. 2017

1,168 citations recorded by the source corpus.

For modeling the 3D world behind 2D images, which 3D representation is most appropriate? A polygon mesh is a promising candidate for its compactness and geometric properties. However, it is not straightforward to model a polygon mesh from 2D images using neural networks because the conversion from a mesh to an image, or rendering, involves a discrete operation called rasterization, which prevents back-propagation. Therefore, in this work, we propose an approximate gradient for rasterization that enables the integration of rendering into neural networks. Using this renderer, we perform single-image 3D mesh reconstruction with silhouette image supervision and our system outperforms the existing voxel-based approach. Additionally, we perform gradient-based 3D mesh editing operations, such as 2D-to-3D style transfer and 3D DeepDream, with 2D supervision for the first time. These applications demonstrate the potential of the integration of a mesh renderer into neural networks and the effectiveness of our proposed renderer.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects (2018, methodology): “[19] replaced the discrete pixel color sampling step with a linear interpolation sampling scheme that admits non-zero gradients.”
Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “Weakly supervised approaches use category-specific priors via 3D shape templates [17, 25, 29] and pose [42] or learn via 2D silhouettes and re-projection on posed views [6,26,34,49].”
Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images (2018, methodology): “(a) Input image; (b) Volume from 3D-R2N2 [6], converted using Marching Cube [21]; (c) Point cloud from PSG [9], converted using ball pivoting [1]; (d) N3MR[17]; (e) Ours; (f) Ground truth.”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “In all of our experiments, we use Neural Mesh Renderer [16] to provide a differentiable implementation of R(·).”
Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, methodology): “Our surface accuracy results can be seen in table 3 compared to [44, 6, 14, 3].”
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (2019, methodology): “Furthermore, current differentiable rasterizertion methods fail to support differentiation with respect to many informative scene properties, such as textures and lighting, leading to low fidelity rendering, and less informative learning signals [14, 20, 21] .”
Fast Interactive Object Annotation With Curve-GCN (2019, methodology): “This has been used previously to optimize 3D mesh vertices to render correctly onto a 2D image [17, 22].”
Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras (2019, background): “Differentiable mesh rendering [26, 16] adopts a geometric approach to occlusions.”
ARCH: Animatable Reconstruction of Clothed Humans (2020, methodology): “Existing approaches can be roughly divided into two categories: mesh rasterization based rendering [9, 19, 25, 29, 43] and volume based rendering [16, 26].”
Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “In the next step of our method, Figure 3b, we recover the object pose using differentiable neural mesh renderer [22], similar to [21, 48, 66].”
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations (2019, methodology): “Discretization-based techniques use voxel grids [7, 16–22], octree hierarchies [23–25], point clouds [11, 26, 27], multiplane images [28], patches [29], or meshes [15, 21, 30, 31].”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “The distance metric d ð(cid:4) ; (cid:4)Þ can be the standard L 2 metric [77], the negative Intersection over Union between the true and reconstructed silhouettes [55], or the binary cross-entropy loss [4], [24].”
Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation (2019, methodology): “With deep learning, many works leverage the data prior to hallucinate the invisible parts, and directly produce shape in 3D volume [5, 11, 48, 13, 34, 41, 18], point cloud [9], mesh models [21], or as an assembling of shape primitive [44, 29].”
Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations (2021, background): “Neural 3D Mesh Renderer Dataset (NMR) [17] .”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, neutral): “To this end, we differentiably render the estimated ﬂow from V t ref to V t ref + k using the Neural Renderer [20].”
Mesh R-CNN (2019, methodology): “N3MR [25] is a weakly supervised approach that ﬁts a mesh via a differentiable ren-derer without 3D supervision.”
Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering (2021, methodology): “We follow the same evaluation protocol as [60] and train a single model across all categories.”
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (2020, methodology): “egmentation as input. Kanazawa et al. [26] utilized adversarial loss to regress plausible SMPL parameters. Baek et al. [5] trained a CNN to estimate parameters of the MANO model using neural renderer [28]. Kolotouros et al. [30] introduced a self-improving system that consists of SMPL parameter regr”
What Do Single-View 3D Reconstruction Networks Learn? (2019, methodology): “Multiple approaches [20, 36, 49, 55, 59] trained shape regressors by comparing projections of ground-truth and predicted shapes.”