Neural Sparse Voxel Fields

By Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-seng Chua, Christian Theobalt. 2020

1,503 citations recorded by the source corpus.

Photo-realistic free-viewpoint rendering of real-world scenes using classical computer graphics techniques is challenging, because it requires the difficult step of capturing detailed appearance and geometry models. Recent studies have demonstrated promising results by learning scene representations that implicitly encode both geometry and appearance without 3D supervision. However, existing approaches in practice often show blurry renderings caused by the limited network capacity or the difficulty in finding accurate intersections of camera rays with the scene geometry. Synthesizing high-resolution imagery from these representations often requires time-consuming optical ray marching. In this work, we introduce Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering. NSVF defines a set of voxel-bounded implicit fields organized in a sparse voxel octree to model local properties in each cell. We progressively learn the underlying voxel structures with a diffentiable ray-marching operation from only a set of posed RGB images. With the sparse voxel octree structure, rendering novel views can be accelerated by skipping the voxels containing no relevant scene content. Our method is over 10 times faster than the state-of-the-art (namely, NeRF) at inference time while achieving higher quality results. Furthermore, by utilizing an explicit sparse voxel representation, our method can easily be applied to scene editing and scene composition. We also demonstrate several challenging tasks, including multi-scene learning, free-viewpoint rendering of a moving human, and large-scale scene rendering.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

  1. Andreas Geiger
  2. Jonathan T. Barron
  3. Matthias Nießner
  4. Gerard Pons-Moll
  5. Michael Niemeyer
  6. Ben Mildenhall
  7. Pratul P. Srinivasan
  8. Andrea Tagliasacchi

In Computer Graphics and Computer-Aided Design

  1. Andreas Geiger
  2. Jonathan T. Barron
  3. Ben Mildenhall
  4. Pratul P. Srinivasan
  5. Matthias Nießner
  6. Peter Hedman
  7. Anpei Chen
  8. Matthew Tancik

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Jitendra Malik
  3. Dacheng Tao
  4. Ming-Hsuan Yang
  5. Andrea Vedaldi
  6. Andreas Geiger
  7. Trevor Darrell
  8. Xiaogang Wang

In 3D Vision & Pose

  1. Andreas Geiger
  2. Matthias Nießner
  3. Jonathan T. Barron
  4. Jitendra Malik
  5. Michael Niemeyer
  6. Justin Johnson
  7. Angela Dai
  8. Songyou Peng

Citations in the corpus