Deep Marching Cubes: Learning Explicit Surface Representations

By Yiyi Liao, Simon Donné, Andreas Geiger. 2018

297 citations recorded by the source corpus.

Existing learning based solutions to 3D surface prediction cannot be trained end-to-end as they operate on intermediate representations (e.g., TSDF) from which 3D surface meshes must be extracted in a post-processing step (e.g., via the marching cubes algorithm). In this paper, we investigate the problem of end-to-end 3D surface prediction. We first demonstrate that the marching cubes algorithm is not differentiable and propose an alternative differentiable formulation which we insert as a final layer into a 3D convolutional neural network. We further propose a set of loss functions which allow for training our model with sparse point supervision. Our experiments demonstrate that the model allows for predicting sub-voxel accurate 3D shapes of arbitrary topology. Additionally, it learns to complete shapes and to separate an object's inside from its outside even in the presence of sparse and incomplete ground truth. We investigate the benefits of our approach on the task of inferring shapes from 3D point clouds. Our model is flexible and can be combined with a variety of shape encoder and shape inference techniques.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Geology, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Geology

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, methodology): “For a fair comparison, we sample points from the surface of the wa-tertight mesh instead of the original model as ground truth for PSGN [17], Pixel2Mesh [70] and DMC [43].”
Deep Level Sets: Implicit Surface Representations for 3D Shape Inference (2019, background): “This is an important distinction from work such as [21] which is derived from a very speciﬁc choice of algorithm.”
Learning Implicit Fields for Generative Shape Modeling (2018, background): “There have been a variety of 3D shape representations for deep learning of shapes, such as voxel grids [9, 14, 28, 45, 46], octrees [18, 35, 40, 43, 44], multi-view images [29, 39], point clouds [1, 12, 13, 32, 33, 47, 48], geometry images [37, 38], deformable mesh/patches [16, 38, 42, 47], and part”
Implicit Geometric Regularization for Learning Shapes (2020, background): “More generally, researchers have suggested us-ing other RGVFs (Dai et al., 2017; Riegler et al., 2017; Stutz & Geiger, 2018; Liao et al., 2018; Michalkiewicz et al., 2019; Jiang et al., 2019).”
Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction (2020, methodology): “tly regress 3D coordinates of points in the point cloud. Voxels. These methods provide non-parametric shape representation using 3D voxel grids which store either occupancy [52,11] or SDF information [14,46,32], similarly to the traditional techniques discussed above. These methods thus inherit the ”
Neural Unsigned Distance Fields for Implicit Function Learning (2020, methodology): “Input PSGN [24] DMC [45] OccNet [49] IF-Net [14] Ours GT from ShapeNet [12], see Fig.”
Texture Fields: Learning Texture Representations in Function Space (2019, background): “In recent years, 3D reconstruction of shapes from single images has made great progress [6, 7, 9, 13, 23, 25, 43, 45].”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, background): “To overcome this limitation, Liao et al. [50] introduced the Deep Marching Cubes, an end-to-end trainable network, which predicts explicit surface representations of arbitrary topology.”
HyperCube: Implicit Field Representations of Voxelized 3D Models (2021, background): “3D objects can be represented using different approaches including voxel grids [4, 5, 6, 7], octrees [8, 9, 10, 11], multi-view images [12, 13, 14], point clouds [15, 16, 17, 18, 19], geometry images [20, 21], deformable meshes [5, 21, 22, 19], and part-based structural graphs [23, 24].”
BSP-Net: Generating Compact Meshes via Binary Space Partitioning (2019, methodology): “is section, we cover representative approaches based on the underlying shape representation learned, with a focus on generative models. Grid models. Early approaches generalized 2D convolutions to 3D [6,14,24,49,50], and employed volumetric grids to represent shapes in terms of coarse occupancy func”
Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics (2019, background): “Recently, learning-based approaches for recovering the 3D geometry from various forms of input have shown promising results [13,14,20,25,34,38,46,54,71].”
Learning to Infer Implicit Surfaces without 3D Supervision (2019, background): “Recent works [22, 5, 6, 7, 8, 23] have shown promising results on supervised learning for 3D shape inference based on implicit representations.”
Local Implicit Grid Representations for 3D Scenes (2020, unknown): “Most deep learning studies have investigated object reconstruction, with input either as an RGB/D image [5, 37, 28, 4, 13, 10, 14] or 3D points [29, 26, 22], and yet few have considered learning to reconstruct full scenes.”
Shape As Points: A Differentiable Poisson Solver (2021, background): “Implicit representations, in contrast, represent the shape as a level set of a continuous function over a discretized voxel grid [14, 27, 35, 69] or more recently parameterized as a neural network, typically referred to as neural implicit functions [10,42,52].”
Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis (2021, unknown): “Compared to DMC [31], which also predicts the SDF values and supervises with a surface loss, DMTET achieves much better reconstruction quality since training using the marching tetrahedra layer is more efficient than calculating an expectation over all possible configurations within one grid cell as”
Convolutional Occupancy Networks (2020, background): “mited in terms of the number of points they can handle. Furthermore, they cannot represent topological relations. Meshes: A popular alternative is to directly regress the vertices and faces of a mesh [14,15,18,21,23,44,45] using a neural network. While some of these works require deforming a templat”
Image2Mesh: A Learning Framework for Single Image 3D Reconstruction (2017, background): “Volumetric autoencoders [12,28,29] and generative adversarial networks (GANs) have been proposed [30–32] to learn of the probabilistic latent space of object shapes for object completion, classification and reconstruction.”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, methodology): “Voxelbased representations [11, 39] have a large memory footprint, which critically limits the output resolution to coarse shapes, without detail.”
MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers (2023, background): “However, like point clouds, these methods require mesh conversion [12, 32, 36, 49, 53] for downstream applications, often leading to dense meshes that don’t capture the properties of the underlying datasets ( e.g ., edge lengths, dihedral angles).”