Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction
By Shichen Liu, Weikai Chen, Tianye Li, Hao Li. 2019
100 citations recorded by the source corpus.
Rendering is the process of generating 2D images from 3D assets, simulated in a virtual environment, typically with a graphics pipeline. By inverting such renderer, one can think of a learning approach to predict a 3D shape from an input image. However, standard rendering pipelines involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence suitable for learning. We present the first non-parametric and truly differentiable rasterizer based on silhouettes. Our method enables unsupervised learning for high-quality 3D mesh reconstruction from a single image. We call our framework `soft rasterizer' as it provides an accurate soft approximation of the standard rasterizer. The key idea is to fuse the probabilistic contributions of all mesh triangles with respect to the rendered pixels. When combined with a mesh generator in a deep neural network, our soft rasterizer is able to generate an approximated silhouette of the generated polygon mesh in the forward pass. The rendering loss is back-propagated to supervise the mesh generation without the need of 3D training data. Experimental results demonstrate that our approach significantly outperforms the state-of-the-art unsupervised techniques, both quantitatively and qualitatively. We also show that our soft rasterizer can achieve comparable results to the cutting-edge supervised learning method and in various cases even better ones, especially for real-world data.
Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Andreas Geiger
- Songyou Peng
- Christian Theobalt
- Sanja Fidler
- Lingjie Liu
- Justin Johnson
- Lior Yariv
- Georgia Gkioxari
In Computer Graphics and Computer-Aided Design
- Andreas Geiger
- Songyou Peng
- Christian Theobalt
- Jonathan T. Barron
- Lingjie Liu
- Sanja Fidler
- Justin Johnson
- Georgia Gkioxari
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Andrea Vedaldi
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Andreas Geiger
In 3D Vision & Pose
- Justin Johnson
- Georgia Gkioxari
- Christian Theobalt
- Marc Pollefeys
- Andreas Geiger
- Richard Szeliski
- David Novotný
- Songyou Peng
Citations in the corpus
- Neural Radiance Flow for 4D View Synthesis and Video Processing (2020, unknown): “Recently, neural continuous implicit fields [9, 17, 35, 41, 49, 51, 58, 61, 75] have been developed to address the discretization issues and limited resolution of classical 3D representations such as voxel grids [5, 11, 40, 56, 70, 71, 74], point clouds [13, 14, 54, 55] and meshes [18, 26, 28, 34, 6”
- Neural Sparse Voxel Fields (2020, methodology): “, 2018) and its follow-up works (Nguyen-Phuoc et al., 2019; Liu et al., 2019c) use a CNN-based decoder for differentiable rendering to render a scene represented as coarse-grained voxel grids.”
- SynSin: End-to-End View Synthesis From a Single Image (2019, background): “This is inspired by [34], which introduces a differentiable renderer for meshes by similarly softening the hard rasterisation decisions and [77] which renderers point clouds by splatting points to a region and accumulating.”
- Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation (2020, methodology): “Liu et al. (2019) computes the gradients analytically by softly assigning contribution of each triangle face to a pixel inmeshbased representations.”
- Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (2019, methodology): “Furthermore, current differentiable rasterizertion methods fail to support differentiation with respect to many informative scene properties, such as textures and lighting, leading to low fidelity rendering, and less informative learning signals [14, 20, 21] .”
- Self-supervised Learning of 3D Objects from Natural Images (2019, background): “Therefore, several differentiable projection modules have been developed for voxels [42, 46], meshes [18, 22], point clouds [14], and implicit functions [23, 35].”
- DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing (2019, background): “Various approaches try to solve the discontinuity issue near triangle boundaries by smoothing the loss function or approximating the gradient [20, 34, 24, 3].”
- Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance (2020, background): “Differentiable rendering systems for learning geometry comes (mostly) in two flavors: differentiable rasterization [28, 20, 10, 25, 4], and differentiable ray casting.”
- Accelerating 3D deep learning with PyTorch3D (2019, methodology): “Our rasterizer departs from [31] in three ways to improve efficiency and modularity.”