Multi-View Mesh Reconstruction with Neural Deferred Shading
By Markus Worchel, Rodrigo Diaz, Weiwen Hu, Oliver Schreer, Ingo Feldmann, Peter Eisert. 2022
67 citations recorded by the source corpus.
We propose an analysis-by-synthesis method for fast multi-view 3D reconstruction of opaque objects with arbitrary materials and illumination. State-of-the-art methods use both neural surface representations and neural rendering. While flexible, neural surface representations are a significant bottleneck in optimization runtime. Instead, we represent surfaces as triangle meshes and build a differentiable rendering pipeline around triangle rasterization and neural shading. The renderer is used in a gradient descent optimization where both a triangle mesh and a neural shader are jointly optimized to reproduce the multi-view images. We evaluate our method on a public 3D reconstruction dataset and show that it can match the reconstruction accuracy of traditional baselines and neural approaches while surpassing them in optimization runtime. Additionally, we investigate the shader and find that it learns an interpretable representation of appearance, enabling applications such as 3D material editing.
Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Justin Johnson
- Georgia Gkioxari
- Rundi Wu
- Jitendra Malik
- Sergey Zakharov
- Basile Van Hoorick
- Carl Vondrick
- Christoph Feichtenhofer
In Computer Graphics and Computer-Aided Design
- Justin Johnson
- Georgia Gkioxari
- Rundi Wu
- Jitendra Malik
- Sergey Zakharov
- Basile Van Hoorick
- Carl Vondrick
- Ruoshi Liu
In 3D Vision & Pose
- Georgia Gkioxari
- Justin Johnson
- Jitendra Malik
- Carl Vondrick
- Christoph Feichtenhofer
- Chaozheng Wu
- Rundi Wu
- Angjoo Kanazawa
Citations in the corpus
- Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “Supervised approaches predict voxels [67, 74] or meshes [69, 71] by training deep nets.”
- Zero-1-to-3: Zero-shot One Image to 3D Object (2023, background): “One line of work builds priors from relying on collections of 3D primitives represented as meshes [58, 62], voxels [17, 60], or point clouds [13, 31], and use image encoders for conditioning.”
- Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, unknown): “Instead of optimizing a 3D representation, alternative approaches utilize category-specific shape priors to regress 3D representations (e.g., point clouds [15, 70], voxels [19, 69], meshes [63, 67], and NeRF [77]) from single images.”