Learning 3D Shape Completion from Laser Scan Data with Weak Supervision

By David Stutz, Andreas Geiger. 2018

254 citations recorded by the source corpus.

3D shape completion from partial point clouds is a fundamental problem in computer vision and computer graphics. Recent approaches can be characterized as either data-driven or learning-based. Data-driven approaches rely on a shape model whose parameters are optimized to fit the observations. Learning-based approaches, in contrast, avoid the expensive optimization step and instead directly predict the complete shape from the incomplete observations using deep neural networks. However, full supervision is required which is often not available in practice. In this work, we propose a weakly-supervised learning-based approach to 3D shape completion which neither requires slow optimization nor direct supervision. While we also learn a shape prior on synthetic data, we amortize, i.e., learn, maximum likelihood fitting using deep neural networks resulting in efficient shape completion without sacrificing accuracy. Tackling 3D shape completion of cars on ShapeNet [5] and KITTI [18], we demonstrate that the proposed amortized maximum likelihood approach is able to compete with a fully supervised baseline and a state-of-the-art data-driven approach while being significantly faster. On ModelNet [49], we additionally show that the approach is able to generalize to other object categories as well.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, methodology): “Voxel Representations: Due to their simplicity, voxels are the most commonly used representation for discriminative [45,55,63] and generative [9,23,58,64,75,77] 3D tasks.”
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation (2019, methodology): “The ability of auto-encoders as a feature learning tool has been evidenced by the vast variety of 3D shape learning works in the literature [16, 49, 2, 22, 55] who adopt auto-encoders for representation learning.”
TopNet: Structural Point Cloud Decoder (2019, background): “Volumetric 3D shape completion: Earlier and some recent works on shape completion leveraging neural networks favored voxel grids, and distance fields representations [6, 15, 29, 27, 18] which are well suited for processing with 3D convolutional neural networks.”
Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction (2020, methodology): “tly regress 3D coordinates of points in the point cloud. Voxels. These methods provide non-parametric shape representation using 3D voxel grids which store either occupancy [52,11] or SDF information [14,46,32], similarly to the traditional techniques discussed above. These methods thus inherit the ”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, background): “To smooth out noise, it is possible to represent shapes as Truncated Signed Distance functions [12] for learning [14, 36, 58, 64].”
Implicit Geometric Regularization for Learning Shapes (2020, background): “…more traditional way of representing surfaces via implicit functions deﬁned on volumetric grids (Wu et al., 2016; Choy et al., 2016; Dai et al., 2017; Stutz & Geiger, 2018), neural implicit representations have the ben-eﬁt of relating the degrees of freedom of the network (i.e., Preliminary work.”
GRNet: Gridding Residual Network for Dense Point Cloud Completion (2020, methodology): “Since the convolution can not be directly applied to point clouds due to their irregularity and unorderedness, most of the existing methods [2,3,4,5,6,7,8] voxelize the point cloud into binary voxels, where 3D convolutional neural networks can be applied.”
PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers (2021, methodology): “Traditional methods for 3D shape completion tasks often adopt voxel grids or distance fields to describe 3D objects [6, 15, 31].”
Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics (2019, background): “They can broadly be categorized into voxel-based [10, 14, 21, 53, 54, 59, 74], point cloud-based [1, 20], mesh-based [25, 29, 34, 71], and continuous representations [13, 22, 28, 38, 39, 46, 56, 72].”
PCN: Point Completion Network (2018, methodology): “While most existing learning-based methods [9, 16, 44, 47, 50, 54, 56, 59] represents shapes using voxels, which are convenient for convolutional neural networks, our method uses point clouds, which preserve complete geometric information about the shapes while being memory efﬁcient.”
SAL: Sign Agnostic Learning of Shapes From Raw Data (2019, methodology): “[34] use variational auto-encoder to learn an implicit surface representations of cars using a volumetric grid.”
Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation (2019, background): “These shapes are then transformed into the world coordinate and converted into signed distance function (SDF) [7, 37, 36].”
ShapeFormer: Transformer-based Shape Completion via Sparse Representation (2022, background): “Since voxels can be processed or generated easily through 3D convolutions thanks to their regularity, they are commonly used in earlier works [18, 20, 32, 59].”
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019, methodology): “the representation they use as voxel-based [8, 13, 19, 61, 64, 73, 82, 83], pointbased [2, 17, 31, 40, 77, 85], mesh-based [24, 32, 41, 55, 80], or implicit representations [3, 12, 22, 30, 48, 49, 56, 66, 81].”
Neural Unsigned Distance Fields for Implicit Function Learning (2020, methodology): “Replacing occupancy with Truncated Signed Distance functions [17] for learning [19, 42, 65, 70] can reduce quantization artifacts, nonetheless TSDF values need to be stored in a grid of fixed limited resolution.”