DeepVoxels: Learning Persistent 3D Feature Embeddings

By Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, Michael Zollhöfer. 2018

732 citations recorded by the source corpus.

In this work, we address the lack of 3D understanding of generative neural networks by introducing a persistent 3D feature embedding for view synthesis. To this end, we propose DeepVoxels, a learned representation that encodes the view-dependent appearance of a 3D scene without having to explicitly model its geometry. At its core, our approach is based on a Cartesian 3D grid of persistent embedded features that learn to make use of the underlying 3D scene structure. Our approach combines insights from 3D geometric computer vision with recent advances in learning image-to-image mappings based on adversarial loss functions. DeepVoxels is supervised, without requiring a 3D reconstruction of the scene, using a 2D re-rendering loss and enforces perspective and multi-view geometry in a principled manner. We apply our persistent 3D scene representation to the problem of novel view synthesis demonstrating high-quality results for a variety of challenging scenes.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, background): “Re-construction by analysis [11] or synthesis via volume rendering [24] of implicit [37, 82] and explicit [33, 56] representations have shown to produce strong results.”
StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, methodology): “Current methods tackle speci c sub-problems like novel view synthesis (Hedman et al. 2018; Sitzmann et al. 2019; ies et al. 2020), relighting under novel lighting conditions (Guo et al. 2019; Xu et al. 2018), animating faces (Fried et al. 2019; Kim et al. 2018; ies et al. 2019), and animating…”
Pixel-aligned Volumetric Avatars (2021, background): “Methods such as Scene Representation Networks (SRNs) [25], which generate a set of weights from a global image encoding (i.”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, methodology): “By introducing differentiable 3D projection and using trainable layers that operate in the 3D and 2D feature space, several recent methods [4, 18, 42, 43, 56, 62] are able to model the geometry and appearance of 3D scenes from 2D images.”
NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction (2021, methodology): “This representation has been applied successfully to shape representation [26, 27, 31, 4, 1, 10, 47, 32], novel view synthesis [38, 23, 15, 28, 21, 33, 34, 40, 37] and multi-view 3D reconstruction [46, 29, 17, 12, 22].”
Immersive light field video with a layered mesh representation (2020, unknown): “Several other approaches [Sitzmann et al. 2019; Thies et al. 2019] learn a deep representation for visual information in the scene.”
IBRNet: Learning Multi-View Image-Based Rendering (2021, methodology): “With advances in differentiable rendering methods [7, 24, 30, 33, 34, 40], many meth-ods [32, 35, 40, 48, 50, 51, 52, 56, 62] showed the ability to learn both scene geometry and appearance from multi-view observations (though only for simple geometry and diffuse materials).”
Neural Sparse Voxel Fields (2020, methodology): “on of only p(z) while c takes both p(z) and v as inputs to model view-dependent color. Different rendering strategies to evaluate this integral are feasible. Surface Rendering. Surface-based methods (Sitzmann et al., 2019b; Liu et al., 2019b; Niemeyer et al., 2019) assume !(p(z)) to be the Dirac fun”
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans (2020, unknown): “In these works, deep neural networks are employed to learn scene representations from 2D images with differentiable renderers, such as voxels [57, 37], point clouds [64, 1], textured meshes [61, 34, 32], multi-plane images [67, 16], and implicit functions [58, 35, 47, 44, 33].”
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations (2019, background): “Some prior work infers voxel grid representations of 3D scenes from images [6, 8, 9] or uses them for 3D-structure-aware generative models [10, 36].”
Neural RGB-D Surface Reconstruction (2021, methodology): “To reduce artifacts from classical reconstruction methods, a series of methods was proposed that use learned spatial priors to predict depth maps from color images [24, 28, 39], to learn multiview stereo using 3D CNNs on voxel grids [34, 68, 82], or multi-plane images [22], to reduce the influence o”
Shape As Points: A Differentiable Poisson Solver (2021, background): “Shape representations are central to many of the recent advancements in 3D computer vision and computer graphics, ranging from neural rendering [43,47,50,58,61] to shape reconstruction [10,28,42, 49,52,54,73].”
Plenoxels: Radiance Fields without Neural Networks (2021, background): “Several papers extended this idea to model a 3D scene using only calibrated 2D image supervision via a differentiable volume rendering formulation [22,28,45,46].”
Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering (2021, unknown): “In these works, deep neural networks are employed to learn novel view synthesis from 2D images in various approaches, such as voxels [25, 15], point clouds [1, 20], textured meshes [27, 13, 11], multi-plane images [27, 31, 6], and implicit functions [26, 14, 18].”
Neural 3D Video Synthesis from Multi-view Video (2021, unknown): “[51] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer.”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, background): “Recent neural scene representations learn low-dimensional nonlinear representations of meshes [31, 66], point clouds [2, 41, 71], sphere sets [27], and dense volumetric grids [32, 58].”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, background): “1 demonstrate that the tri-plane representation is capable of representing this complex scene, albeit without view-dependent effects, outperforming dense feature volume representations [38, 61] and fully implicit representations [45] in terms of PSNR and SSIM, while offering considerable advantages ”
Advances in Neural Rendering (2021, methodology): “In contrast, DeepVoxels [STH ∗ 19] optimizes a 3D voxel grid of features jointly with a learned ren-derer using images of a single scene, without requiring any external training data.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “Regular structures such as voxelgrids or octrees [13, 48, 14, 88, 94, 89, 43, 81, 107, 59, 60] or unsupervised decomposition [73, 85, 109, 83] enable editability via manipulation of localized parameters.”