Learning to Infer Implicit Surfaces without 3D Supervision

By Shichen Liu, Shunsuke Saito, Weikai Chen, Hao Li. 2019

213 citations recorded by the source corpus.

Recent advances in 3D deep learning have shown that it is possible to train highly effective deep models for 3D shape generation, directly from 2D images. This is particularly interesting since the availability of 3D models is still limited compared to the massive amount of accessible 2D images, which is invaluable for training. The representation of 3D surfaces itself is a key factor for the quality and resolution of the 3D output. While explicit representations, such as point clouds and voxels, can span a wide range of shape variations, their resolutions are often limited. Mesh-based representations are more efficient but are limited by their ability to handle varying topologies. Implicit surfaces, however, can robustly handle complex shapes, topologies, and also provide flexible resolution control. We address the fundamental problem of learning implicit surfaces for shape inference without the need of 3D supervision. Despite their advantages, it remains nontrivial to (1) formulate a differentiable connection between implicit surfaces and their 2D renderings, which is needed for image-based supervision; and (2) ensure precise geometric properties and control, such as local smoothness. In particular, sampling implicit surfaces densely is also known to be a computationally demanding and very slow operation. To this end, we propose a novel ray-based field probing technique for efficient image-to-field supervision, as well as a general geometric regularizer for implicit surfaces, which provides natural shape priors in unconstrained regions. We demonstrate the effectiveness of our framework on the task of single-view image-based 3D shape digitization and show how we outperform state-of-the-art techniques both quantitatively and qualitatively.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In 3D Vision & Pose

Citations in the corpus

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, background): “…volume density [27], occupancy [24], and signed distance [32], and have achieved state-of-the-art results across a variety of tasks such as shape representation [9, 10, 12, 13, 17, 26, 32], texture synthesis [15, 31], shape inference from images [22, 23], and novel view synthesis [27, 29, 35, 37].”
pixelNeRF: Neural Radiance Fields from One or Few Images (2020, methodology): “However, many of these methods [50, 45, 21, 22, 28] require object masks; in contrast, pixelNeRF can be trained from images alone, allowing it to be applied to scenes of two objects without modification.”
ARCH: Animatable Reconstruction of Clothed Humans (2020, background): “Existing approaches can be roughly divided into two categories: mesh rasterization based rendering [9, 19, 25, 29, 43] and volume based rendering [16, 26].”
Advances in Neural Rendering (2021, background): “An implicit differentiable renderer [SZW19, NMOG20,LZP ∗ 20,LSCL19,YKM ∗ 20,KJJ ∗ 21,BKW21,TLY ∗ 21] can be implemented by ﬁrst ﬁnding the intersection between a viewing ray and the surface using the geometric branch of the neural implicit function, and then obtaining the RGB value of this point…”
D-NeRF: Neural Radiance Fields for Dynamic Scenes (2020, background): “For instance, [20] proposed an efﬁcient ray-based ﬁeld probing algorithm for efﬁcient image-to-ﬁeld supervision.”
Convolutional Occupancy Networks (2020, methodology): “Implicit models have been adopted for learning implicit representations from images [23, 24, 29, 41], for encoding texture information [30], for 4D reconstruction [28] as well as for primitive-based reconstruction [10, 11, 15, 32].”
PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization (2020, methodology): “Future work may include incorporating human speciﬁc priors (e.g., semantic segmentation, pose, and parametric 3D face models) and adding 2D supervision of implicit surface [37, 25] to further support in-the-wild inputs.”
Multi-View Mesh Reconstruction with Neural Deferred Shading (2022, unknown): “Surfaces are often represented with implicit functions or more specifically implicit neural representations [33, 41, 44] where the indicator function is modeled by a multi-layer perceptron (MLP) or any other form of neural network and optimized with the rendering networks in an end-to-end fashion.”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “To overcome the need of 3D supervision, several works [50, 51, 66, 81, 92] propose differentiable rendering techniques.”
Learned Initializations for Optimizing Coordinate-Based Neural Representations (2020, methodology): “Later work combined this idea with various formulations of differentiable rendering to recover neural representations of 3D shape using only 2D image observations [18, 19, 24, 26, 34, 37].”
Neural Sparse Voxel Fields (2020, methodology): “…point (Michalkiewicz et al., 2019; Mescheder et al., 2019; Chen and Zhang, 2019; Park et al., 2019; Peng et al., 2020), while other works learn 3D representations only from images with differentiable renderers (Liu et al., 2019d; Saito et al., 2019, 2020; Niemeyer et al., 2019; Jiang et al., 2020).”
Editing Conditional Radiance Fields (2021, background): “Instead, several recent approaches implicitly encode a continuous volumetric representation of shape [14, 21, 38, 39, 42, 43, 48, 50, 54, 61] or both shape and view-dependent appearance [41, 45, 47, 59, 63, 66, 69, 76, 12, 73] in the weights of a neural network.”
RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs (2021, background): “Several works [29, 37, 41, 52, 61] proposed differentiable rendering approaches to learn neural representations from only multiview image supervision.”
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019, background): “[45] propose to infer implicit representations from multi-view silhouettes by performing max-pooling over the intersections of rays with a sparse number of supporting regions.”
DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing (2019, background): “[26] firstly propose to learn from 2D observations over occupancy networks [29].”
Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance (2020, methodology): “Thus far, differentiable rendering systems with implicit neural representations [26, 27, 33] did not incorporate lighting and reﬂectance properties required for producing faithful appearance of 3D geometry in images, nor did they deal with trainable camera locations and orientations.”