HyperNeRF

By Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Ricardo Martin-Brualla, S. M. Seitz. ACM Transactions on Graphics, 2021

337 citations recorded by the source corpus.

Neural Radiance Fields (NeRF) are able to reconstruct scenes with unprecedented fidelity, and various recent works have extended NeRF to handle dynamic scenes. A common approach to reconstruct such non-rigid scenes is through the use of a learned deformation field mapping from coordinates in each input image into a canonical template coordinate space. However, these deformation-based approaches struggle to model changes in topology, as topological changes require a discontinuity in the deformation field, but these deformation fields are necessarily continuous. We address this limitation by lifting NeRFs into a higher dimensional space, and by representing the 5D radiance field corresponding to each individual input image as a slice through this "hyper-space". Our method is inspired by level set methods, which model the evolution of surfaces as slices through a higher dimensional surface. We evaluate our method on two tasks: (i) interpolating smoothly between "moments", i.e., configurations of the scene, seen in the input images while maintaining visual plausibility, and (ii) novel-view synthesis at fixed moments. We show that our method, which we dub HyperNeRF, outperforms existing methods on both tasks. Compared to Nerfies, HyperNeRF reduces average error rates by 4.1% for interpolation and 8.6% for novel-view synthesis, as measured by LPIPS. Additional videos, results, and visualizations are available at hypernerf.github.io.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/875129e6a54f18a537ba34845fda03701cacf388

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

I M Avatar: Implicit Morphable Head Avatars from Videos (2021, methodology): “Previous works either achieve this by conditioning the implicit representation on expressions [18] or via a separate displacement-based warping field [34, 35].”
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “[26,13,25] alternatively adopt windowed positional encoding to aid learning dynamic shapes via a coarse-to-fine training.”
TensoRF: Tensorial Radiance Fields (2022, background): “This representation has been quickly extended and applied in diverse graphics and vision applications, including generative models [6,26], appearance acquisition [2,4], surface reconstruction [40,27], fast rendering [33,47,12], appearance editing [42,19], dynamic capture [17,28].”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, methodology): “Benefiting from these advantages, NeRF has been widely used in many fields, such as 3D modeling [53, 58], human face/body digitization [12, 17, 42, 43, 48, 55], generating 4D free-view video [31,39], etc.”
Point-NeRF: Point-based Neural Radiance Fields (2022, methodology): “However, most methods [3, 26, 38, 53] still follow the original NeRF framework and train per-scene MLPs to represent radiance fields.”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, methodology): “A series of subsequent NeRF works [Li et al. 2021a; Park et al. 2021a,b; Pumarola et al. 2021; Tretschk et al. 2021a] improve radiance field construction towards dynamic scenes.”