Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning

By Shichen Liu, Tianye Li, Weikai Chen. IEEE International Conference on Computer Vision, 2019

812 citations recorded by the source corpus.

Rendering bridges the gap between 2D vision and 3D scenes by simulating the physical process of image formation. By inverting such renderer, one can think of a learning approach to infer 3D information from 2D images. However, standard graphics renderers involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence able to be learned. Unlike the state-of-the-art differentiable renderers, which only approximate the rendering gradient in the back propagation, we propose a truly differentiable rendering framework that is able to (1) directly render colorized mesh using differentiable functions and (2) back-propagate efficient supervision signals to mesh vertices and their attributes from various forms of image representations, including silhouette, shading and color images. The key to our framework is a novel formulation that views rendering as an aggregation function that fuses the probabilistic contributions of all mesh triangles with respect to the rendered pixels. Such formulation enables our framework to flow gradients to the occluded and far-range vertices, which cannot be achieved by the previous state-of-the-arts. We show that by using the proposed renderer, one can achieve significant improvement in 3D unsupervised single-view reconstruction both qualitatively and quantitatively. Experiments also demonstrate that our approach is able to handle the challenging tasks in image-based shape fitting, which remain nontrivial to existing differentiable renderers. Code is available at https://github.com/ShichenLiu/SoftRas.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8b751405526c28245eea5e925a6ede034c287bdb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Modular primitives for high-performance differentiable rendering (2020, methodology): “As comparison methods we used the official implementation of Soft Rasterizer [Liu et al. 2019], and PyTorch3D [Ravi et al.”
Mixture of volumetric primitives for efficient neural rendering (2021, background): “Differentiable rasterization approaches [Chen et al. 2019; Kato et al. 2018; Laine et al. 2020; Liu et al. 2019; Loper and Black 2014; Petersen et al. 2019; Ravi et al. 2020; Valentin et al. 2019] enable the end-to-end integration of deep neural networks with this classical computer graphics…”
Learning Smooth Neural Functions via Lipschitz Regularization (2022, methodology): “Several methods (e.g., [Hertz et al. 2020; Liu et al. 2019; Wang et al. 2018]) use Laplacian regularization which penalizes the difference between a vertex and the center of mass of its 1-ring neighbors.”
Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, methodology): “This approach, successfully utilized in [54, 32] for synthetic datasets where the training shapes are approximately rigidly aligned, is however insufficient when facing more challenging in-the-wild scenarios.”
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, background): “More recently, the joint reconstruction of geometry and materials is proposed given calibrated multi-view images [6, 7, 19, 48].”
Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations (2021, result): “These meth-ods have demonstrated favorable quantitative and qualitative results in comparison to other previous work [41, 37, 28, 22].”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, background): “, need to design various approximation methods [19, 23, 27, 28, 32] to alleviate the non-differentiable problems of their rendering process.”
Differentiable vector graphics rasterization for editing and learning (2020, background): “Differentiating 3D rendering has drawn attention in both computer graphics and vision research recently (e.g., [de La Gorce et al. 2011; Kato et al. 2018; Liu et al. 2018, 2019; Loper and Black 2014]).”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, background): “It also enables optimization within the bounds of the learned prior [14, 19, 28] and weak-supervision when integrated in a differentiable form [30] into neural training processes [5, 23, 26, 45, 48, 68].”
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (2019, methodology): “Furthermore, current differentiable rasterizertion methods fail to support differentiation with respect to many informative scene properties, such as textures and lighting, leading to low fidelity rendering, and less informative learning signals [14, 20, 21] .”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, background): “Several soft relaxations have thus been proposed [35, 29, 32].”
pixelNeRF: Neural Radiance Fields from One or Few Images (2020, methodology): “2), we also include grayscale rendering of SoftRas [21] results.”
3D Morphable Face Models—Past, Present, and Future (2019, neutral): “Liu et al. [2019a] propose a rasterizer in which triangles make a soft (and hence differentiable) contribution to image appearance.”
Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches (2021, methodology): “We extend the traditional encoder-decoder architecture used in some recent works [20, 22].”
Learning to Infer Implicit Surfaces without 3D Supervision (2019, methodology): “To exploit the abundance of image datasets, various differentiable rendering techniques [1, 2, 3, 4] were introduced recently, to learn 3D generative models directly from massive amounts of 2D pictures.”
Advances in Neural Rendering (2021, methodology): “This has been implemented, for example, in [LLCL19,RRN ∗ 20].”
ARCH: Animatable Reconstruction of Clothed Humans (2020, methodology): “SoftRasterizer [25] and DIB-R [9], in contrast, redefine the rasterization as a continuous and differentiable function, allowing gradients to be computed automatically.”
Kubric: A scalable dataset generator (2022, methodology): “We observe that this processed dataset allows us to train a SOFTRAS capable of reconstructing a wider range of objects than in the original work of Liu et al. [58] but the performance for some classes are poor, which will hopefully inspire further research into more powerful and 2D-to-3D reconstruct”
HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction (2022, methodology): “We utilize a differentiable renderer SoftRas [31] and auxiliary loss terms in HOnnotate [20] to optimize the pose by gradient descent.”
Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (2021, methodology): “Early methods [18, 70, 58, 61, 40] predicted a global scene encoding z global = Φ CNN ( I src ) with a deep convolutional network Φ CNN that solely analyzed the colors of source image pixels.”