Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

By Wenzheng Chen, Jun Gao, Huan Ling, Edward James Smith, Jaakko Lehtinen, Alec Jacobson, Sanja Fidler. 2019

398 citations recorded by the source corpus.

Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting with light, in a process called rendering. Enabling ML models to understand image formation might be key for generalization. However, due to an essential rasterization step involving discrete assignment operations, rendering pipelines are non-differentiable and thus largely inaccessible to gradient-based ML techniques. In this paper, we present {\emph DIB-R}, a differentiable rendering framework which allows gradients to be analytically computed for all pixels in an image. Key to our approach is to view foreground rasterization as a weighted interpolation of local properties and background rasterization as a distance-based aggregation of global geometry. Our approach allows for accurate optimization over vertex positions, colors, normals, light directions and texture coordinates through a variety of lighting models. We showcase our approach in two ML applications: single-image 3D object prediction, and 3D textured object generation, both trained using exclusively using 2D supervision. Our project website is: this https URL

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “Weakly supervised approaches use category-specific priors via 3D shape templates [17, 25, 29] and pose [42] or learn via 2D silhouettes and re-projection on posed views [6,26,34,49].”
Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation (2020, background): “Wu et al. (2016a), Kato and Harada (2018), Henderson and Ferrari (2018) and Chen et al. (2019) use image based annotations like silhouettes, 2D keypoints or object masks.”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, neutral): “While Chamfer distance has been widely used in the field of 3D generative models and reconstruction [8, 15, 26], LFD has received a lot attention in computer graphics [10].”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, methodology): “3DM, T, VP from I Birds [7] I, 2DS Pred.”
Accelerating 3D deep learning with PyTorch3D (2019, background): “Through careful design of these components, we improve efﬁciency and modularity compared to prior differentiable renderers [25, 31, 7].”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, background): “…in the form of point clouds (Wu et al., 2016; Gadelha et al., 2017; Smith & Meger, 2017), triangle meshes (Gao et al., 2022; Pavllo et al., 2020; Chen et al., 2019; Luo et al., 2021) or volumes (Chan et al., 2022; Or-El et al., 2022; Bergman et al., 2022; Skorokhodov et al., 2022) Diffusion…”
Objaverse-XL: A Universe of 10M+ 3D Objects (2023, methodology): “Here, several methods explore novel representations [35, 36, 37, 38], network architectures [39, 40] and differentiable rendering techniques [41, 42, 43, 44, 45] to predict the 3D shapes and textures of objects from images with or without 3D supervision.”
ARCH: Animatable Reconstruction of Clothed Humans (2020, methodology): “Existing approaches can be roughly divided into two categories: mesh rasterization based rendering [9, 19, 25, 29, 43] and volume based rendering [16, 26].”
Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis (2021, neutral): “Surface-based Methods directly predict triangular meshes and have achieved impressive results for reconstructing and synthesizing simpler shapes [54, 23, 5, 41, 7].”
DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing (2019, background): “Various approaches try to solve the discontinuity issue near triangle boundaries by smoothing the loss function or approximating the gradient [21, 36, 25, 3].”
iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020, background): “Differentiable mesh renderers [2], [24] have also been explored for pose estimation.”
Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance (2020, background): “…33, 24, 26, 38, 27], or rasterization [28, 20, 10, 25, 4], while popular models to represent 3D geometry include point clouds [49], triangle meshes [4], implicit representations deﬁned over volumetric grids [17], and recently also neural implicit representations, namely, zero level sets of neural…”
Pixel-aligned Volumetric Avatars (2021, background): “Efforts in differentiable rasterizers [3, 11, 8, 6] have shown impressive results in generating meshes from single and multi-view images without 3D supervision, but the generated meshes usually have restrictions in terms of topology and fail to capture high frequency details.”
BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images (2020, background): “This can be achieved with explicit 3D representations, like appearance ﬂow [58], occupancy voxel grids [43, 59], meshes, or shape templates [27, 46, 56], in conjunction with handcrafted differentiable renderers [8, 17, 31, 33].”
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, background): “More recently, the joint reconstruction of geometry and materials is proposed given calibrated multi-view images [6, 7, 19, 48].”
Mixture of volumetric primitives for efficient neural rendering (2021, background): “Differentiable rasterization approaches [Chen et al. 2019; Kato et al. 2018; Laine et al. 2020; Liu et al. 2019; Loper and Black 2014; Petersen et al. 2019; Ravi et al. 2020; Valentin et al. 2019] enable the end-to-end integration of deep neural networks with this classical computer graphics represe”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[107] Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler.”