Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction

By Shichen Liu, Weikai Chen, Tianye Li, Hao Li. 2019

100 citations recorded by the source corpus.

Rendering is the process of generating 2D images from 3D assets, simulated in a virtual environment, typically with a graphics pipeline. By inverting such renderer, one can think of a learning approach to predict a 3D shape from an input image. However, standard rendering pipelines involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence suitable for learning. We present the first non-parametric and truly differentiable rasterizer based on silhouettes. Our method enables unsupervised learning for high-quality 3D mesh reconstruction from a single image. We call our framework `soft rasterizer' as it provides an accurate soft approximation of the standard rasterizer. The key idea is to fuse the probabilistic contributions of all mesh triangles with respect to the rendered pixels. When combined with a mesh generator in a deep neural network, our soft rasterizer is able to generate an approximated silhouette of the generated polygon mesh in the forward pass. The rendering loss is back-propagated to supervise the mesh generation without the need of 3D training data. Experimental results demonstrate that our approach significantly outperforms the state-of-the-art unsupervised techniques, both quantitatively and qualitatively. We also show that our soft rasterizer can achieve comparable results to the cutting-edge supervised learning method and in various cases even better ones, especially for real-world data.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

  1. Andreas Geiger
  2. Songyou Peng
  3. Christian Theobalt
  4. Sanja Fidler
  5. Lingjie Liu
  6. Justin Johnson
  7. Lior Yariv
  8. Georgia Gkioxari

In Computer Graphics and Computer-Aided Design

  1. Andreas Geiger
  2. Songyou Peng
  3. Christian Theobalt
  4. Jonathan T. Barron
  5. Lingjie Liu
  6. Sanja Fidler
  7. Justin Johnson
  8. Georgia Gkioxari

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Jitendra Malik
  3. Andrea Vedaldi
  4. Dacheng Tao
  5. Ming-Hsuan Yang
  6. Trevor Darrell
  7. Xiaogang Wang
  8. Andreas Geiger

In 3D Vision & Pose

  1. Justin Johnson
  2. Georgia Gkioxari
  3. Christian Theobalt
  4. Marc Pollefeys
  5. Andreas Geiger
  6. Richard Szeliski
  7. David Novotný
  8. Songyou Peng

Citations in the corpus