Modular primitives for high-performance differentiable rendering

By Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, Timo Aila. ACM Transactions on Graphics, 2020

656 citations recorded by the source corpus.

We present a modular differentiable renderer design that yields performance superior to previous methods by leveraging existing, highly optimized hardware graphics pipelines. Our design supports all crucial operations in a modern graphics pipeline: rasterizing large numbers of triangles, attribute interpolation, filtered texture lookups, as well as user-programmable shading and geometry processing, all in high resolutions. Our modular primitives allow custom, high-performance graphics pipelines to be built directly within automatic differentiation frameworks such as PyTorch or TensorFlow. As a motivating application, we formulate facial performance capture as an inverse rendering problem and show that it can be solved efficiently using our tools. Our results indicate that this simple and straightforward approach achieves excellent geometric correspondence between rendered results and reference imagery.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4424714a85ca14ec2d92a8bd1501085fa1e7a51f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “This is due to the mipmap texture sampling technique used in differentiable rasterization (Laine et al., 2020).”
Multi-View Mesh Reconstruction with Neural Deferred Shading (2022, neutral): “[37].”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[108] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila.”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, methodology): “In the heart of our approach is a generative process that utilizes a differentiable explicit surface extraction method [60] and a differentiable rendering technique [47, 37].”
A Survey on 3D Gaussian Splatting (2024, neutral): “In addition, from a differentiable rendering perspective, 3D GS complements mesh-based differentiable rasterizers such as nvdiffrast [119], which operate on explicit triangle meshes using high-performance modular rasterization primitives.”
CLIP-Mesh: Generating textured meshes from text using pretrained image-text models (2022, methodology): “CLIP takes 224x224 input images but we find that rendering at a larger 512x512 resolution and down scaling to 224x224 improves results, it also plays well with the differentiable render we use [Laine et al. 2020] since it relies on anti aliasing for gradients and rendering at a larger resolution…”
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, methodology): “In this work, We use DMTET as our geometry representation and render the mesh extracted from MT layer iteratively by a differentiable renderer[16, 28].”
Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts (2021, methodology): “For the image-approximation with shapes task, we find that modern ES algorithms offer large improvements in both quality and efficiency when compared to traditional genetic algorithms, and as we will also demonstrate, even comparable to state-of-the-art differentiable rendering methods [29].”
Extracting Triangular 3D Models, Materials, and Lighting From Images (2021, methodology): “We optimize the end-to-end image loss of an explicit mesh representation, supporting intrinsic decomposition of shape, materials and lighting by design, and utilizing efficient differentiable rasterization [30].”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, background): “, need to design various approximation methods [19, 23, 27, 28, 32] to alleviate the non-differentiable problems of their rendering process.”
Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, background): “More recently, Gao et al. (2022) train a GAN that directly produces full 3D meshes, paired with a discriminator that inputs differentiably-rendered (Laine et al., 2020) views of the generated meshes.”
GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians (2023, methodology): “Here, we believe that modeling these parts directly would provide an interesting future research avenue; for instance, recent methods show promising result in reconstructing hair [36, 43].”
Shap-E: Generating Conditional 3D Implicit Functions (2023, methodology): “This INR can be used to construct 3D triangle meshes in a differentiable manner, and the resulting meshes can be rendered efﬁciently using differentiable rasterization libraries [32].”
Structured 3D Latents for Scalable and Versatile 3D Generation (2024, positive): “We utilize Nvdiffrast [16] to render the extracted mesh along with its attributes, producing a foreground mask M , a depth map D , a normal map N m directly derived from the mesh, an RGB image C , and a normal map N from the predicted normals.”
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (2024, positive): “The hybrid representation of DMTet geometry and hash grid [34] textures enables the capture of high-quality 3D information, which can be efficiently rendered using differentiable rasteri-zation [18].”