Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis

By Tianchang Shen, Jun Gao, K. Yin, Ming-Yu Liu, Sanja Fidler. Neural Information Processing Systems, 2021

629 citations recorded by the source corpus.

We introduce DMTet, a deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. It marries the merits of implicit and explicit 3D representations by leveraging a novel hybrid 3D representation. Compared to the current implicit approaches, which are trained to regress the signed distance values, DMTet directly optimizes for the reconstructed surface, which enables us to synthesize finer geometric details with fewer artifacts. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology. The core of DMTet includes a deformable tetrahedral grid that encodes a discretized signed distance function and a differentiable marching tetrahedra layer that converts the implicit signed distance representation to the explicit surface mesh representation. This combination allows joint optimization of the surface geometry and topology as well as generation of the hierarchy of subdivisions using reconstruction and adversarial losses defined explicitly on the surface mesh. Our approach significantly outperforms existing work on conditional shape synthesis from coarse voxel inputs, trained on a dataset of complex 3D animal shapes. Project page: https://nv-tlabs.github.io/DMTet/.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8e970913466a81207230a09f6516bab944563cc0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, methodology): “Furthermore, following practices from DEFTET [15] and DMTET [26], we us two copies of the geometry generator.”
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures (2022, methodology): “The output textured mesh is then obtained in a differentiable manner by utilizing DMTet [42].”
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, methodology): “NVDiffRec [35] combines differentiable marching tetrahedra [42] with differentiable rasterization to perform full inverse rendering and extract triangle meshes, materials, and lighting from images.”
LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching (2023, methodology): “Previously, various representations have been introduce to text-to-3D generations [3, 8, 31, 39, 44].”
Shap-E: Generating Conditional 3D Implicit Functions (2023, background): “• DMTet [56] and its extension GET3D [18] represent a textured 3D mesh as a function mapping coordinates to colors, signed distances, and vertex offsets.”
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, methodology): “• For geometry learning, we use a hybrid representation of DMTET, which supports learning surface deformation via a differentiable mesh extraction; we propose to render and encode the surface normal extracted from DMTET as the input of the pre-trained image diffusion model, which enables more subtle”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, background): “Several concurrent works [53, 61] opt to apply supervisions on the mesh representation directly us-ing differentiable surface optimization techniques [40, 41].”
AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation (2022, neutral): “TSDF or voxels, and it is not obvious whether our autoregressive modeling framework can be adapted to other 3D representations such as meshes or neural implicit functions [26].”
Magic3D: High-Resolution Text-to-3D Content Creation (2022, methodology): “Formally, we represent the 3D shape using a deformable tetrahedral grid (VT , T ), where VT is the vertices in the grid T [8, 39].”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “In the fine stage, we extract a mesh from the trained NeRF using marching cubes [7], and adopt a DMTet [43] representation to finetune the 3D mesh using SDS-based diffusion guidance from SV3D at full-resolution.”
ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation (2023, methodology): “[44] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler.”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, methodology): “To this end, we adopt DMTet [78], which is a hybrid SDF-Mesh representation and is capable of generating high-resolution 3D shapes due to its high memory efficiency.”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, positive): “Furthermore, these methods require a time-intensive, object-specific optimization process to distill diffusion priors [7, 33, 40, 50, 60], combined with monocular cues (depth or normals [14, 38]), into neural 3D representations [43, 58].”
CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (2024, neutral): “…various neural fields [Chen et al. 2024; Huang et al. 2024; Lin et al. 2023; Wu et al. 2024; Yu et al. 2023b; Zhu et al. 2024], ranging from DMTet [Shen et al. 2021] to the most recent 3D Gaussian splatting [Kerbl et al. 2023], Various modifications managed to elevate the performance [Chen et al.…”
MVDream: Multi-view Diffusion for 3D Generation (2023, methodology): “Magic3D performs relatively better by using DMTet Shen et al. (2021) for second stage refinement.”
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (2024, background): “…to effectively distill detailed information from 2D diffusion models into 3D. Due to the significant memory consumption associated with high-resolution rendering of NeRF, Magic3D [22] first converts NeRF to DMTet [42] and subsequently trains a second stage for finer resolution refinement.”
TEXTure: Text-Guided Texturing of 3D Shapes (2023, methodology): “Get3D [16] is trained to generate shape and texture through a DMTet [46] mesh extractor and 2D adversarial losses.”