Neural volumes

By Stephen Lombardi, Tomas Simon, Jason M. Saragih, Gabriel Schwartz, Andreas M. Lehrmann, Yaser Sheikh. ACM Transactions on Graphics, 2019

338 citations recorded by the source corpus.

Modeling and rendering of dynamic scenes is challenging, as natural scenes often contain complex phenomena such as thin structures, evolving topology, translucency, scattering, occlusion, and biological motion. Mesh-based reconstruction and tracking often fail in these cases, and other approaches (e.g., light field video) typically rely on constrained viewing conditions, which limit interactivity. We circumvent these difficulties by presenting a learning-based approach to representing dynamic objects inspired by the integral projection model used in tomographic imaging. The approach is supervised directly from 2D images in a multi-view capture setting and does not require explicit reconstruction or tracking of the object. Our method has two primary components: an encoder-decoder network that transforms input images into a 3D volume representation, and a differentiable ray-marching operation that enables end-to-end training. By virtue of its 3D representation, our construction extrapolates better to novel viewpoints compared to screen-space rendering techniques. The encoder-decoder architecture learns a latent representation of a dynamic scene that enables us to produce novel content sequences not seen during training. To overcome memory limitations of voxel-based representations, we learn a dynamic irregular grid structure implemented with a warp field during ray-marching. This structure greatly improves the apparent resolution and reduces grid-like artifacts and jagged motion. Finally, we demonstrate how to incorporate surface-based representations into our volumetric-learning framework for applications where the highest resolution is required, using facial performance capture as a case in point.

Subfields: Aerospace Engineering, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/31de0799f53e48c166b714f9fce7698a7a4a6c0b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

HyperNeRF (2021, methodology): “We compare our method with Nerfies [Park et al. 2020], Neural Volumes [Lombardi et al. 2019], and NSFF [Li et al. 2020].”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, neutral): “…PSNR↑ MS-SSIM↑ PSNR↑ MS-SSIM↑ PSNR↑ MS-SSIM↑ PSNR↑ MS-SSIM↑ NeRF [Mildenhall et al. 2020] ∼ hours 19.9 0.653 20.7 0.780 19.9 0.777 20.0 0.769 20.1 0.745 NV [Lombardi et al. 2019] ∼ hours 17.7 0.623 16.2 0.665 17.6 0.615 15.9 0.380 16.9 0.571 NSFF [Li et al. 2021a] ∼ hours 26.1 0.871 27.7 0.947…”
SofGAN: A Portrait Image Generator with Dynamic Styling (2020, background): “…2020], reenactment [Cao et al. 2013; Geng et al. 2018; Siarohin et al. 2019; Tewari et al. 2020; Thies et al. 2016], novel views synthesis [Chen et al. 2018; Lombardi et al. 2019; Mildenhall et al. 2020; Sitzmann et al. 2019a,b; Thies et al. 2019; Zhou et al. 2020], etc, even with sparse inputs.”
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “Neural volumetric representations [21,24,16,22,15,37] have demonstrated remarkable capability in representing 3D objects and scenes from images.”
BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2023, methodology): “…sidestep the difficulty of high-quality mesh reconstruction by using volumetric representations of geometry and appearance, such as voxel grids [Lombardi et al. 2019; Penner and Zhang 2017; Szeliski and Golland 1999; Vo-giatzis et al. 2007] or multiplane images [Srinivasan et al. 2019;…”