Extracting Triangular 3D Models, Materials, and Lighting From Images

By Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, T. Muller, Sanja Fidler. Computer Vision and Pattern Recognition, 2021

511 citations recorded by the source corpus.

We present an efficient method for joint optimization of topology, materials and lighting from multi-view image observations. Unlike recent multi-view reconstruction approaches, which typically produce entangled 3D representations encoded in neural networks, we output triangle meshes with spatially-varying materials and environment lighting that can be deployed in any traditional graphics engine unmodified. We leverage recent work in differentiable rendering, coordinate-based networks to compactly represent volumetric texturing, alongside differentiable marching tetrahedrons to enable gradient-based optimization directly on the surface mesh. Finally, we introduce a differentiable formulation of the split sum approximation of environment lighting to efficiently recover all-frequency lighting. Experiments show our extracted models used in advanced scene editing, material decomposition, and high quality view interpolation, all running at interactive rates in triangle-based renderers (rasterizers and path tracers).

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5d277f165a108d585262955c648ff2224955d9c0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, unknown): “Traditional primitive-based representations like meshes and points [6, 26, 32, 55] are faster due to the use of rasterization techniques tailored for modern GPUs.”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, methodology): “2021] or completely [Chen et al. 2023; Munkberg et al. 2022b] to a surface-like representation degrades image quality but results in compact representations that are cheap to render.”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, methodology): “Regularization To remove internal floating faces that are not visible in any of the views, we further regularize the geometry generator with a cross-entropy loss defined between the SDF values of the neighboring vertices [47]:”
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, methodology): “NVDiffRec [35] combines differentiable marching tetrahedra [42] with differentiable rasterization to perform full inverse rendering and extract triangle meshes, materials, and lighting from images.”
Magic3D: High-Resolution Text-to-3D Content Creation (2022, methodology): “During optimization, we render the extracted surface mesh into high-resolution images using a differentiable rasterizer [19,28].”
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, background): “One approach is to impose physics-based inductive biases in its optimization process to enable changes in materials or scene lighting [44, 1, 40, 27, 25].”
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, methodology): “Following [28], the hemisphere integration can be calculated using the split-sum method.”